[Software] Brickficiency - Find the cheapest BL stores to buy parts

[tsrplatelayer]

Thanks - but I already had understood that issue - all parts are 'U'.

I have found that by ensuring that I completely exit and re-load BF and then slowly adding parts, say 5 or 10 at a time, all works well, if prices cannot be found the scan stops and I can see which part caused the problem. Remebering to exit BF, modify the partys list with BS and then restart BF and re-open the file.

(If I tried just closing the file in BF. Modifying it with BL and then reopening it with BF without restarting BF there were issues.)

I then take that part out and try again, When all the parts I added are 'priced' the process complets. I can then take out any parts that are 'scarce' (<10 shops)

I then sat 'exclude' on all those parts and add another 10.

Once the process was complete for all 100ish parts I was left with 2 lists, one that was able to find parts in more than 10 shops - 65 parts, 110 items, and one with scarceparts.

Sadly no 4 shop solutions in the UK.

The scarce list has 35 lots, 5 items, of those 12 lots have no 6 month average price from BL.

I suspect it was the last 12 lots that were causing the problem.

Now I understand whats happening and I can see a process to use.

1. Build the list in Brickstock - populate with prices and exclude those with no price available.

2. Upload the remainder to a BL Want file and look at availability within the required region, exclude scarce parts

3. Run BF on the remainder.

Allan

Edited July 6, 2015 by tsrplatelayer

[S_Bartfast]

That does seem odd.

I've only been using Brickficiency for a short time myself and have had a few odd experiences similar to what you describe but like you was able to find workarounds.

I've mainly imported lists from Lego Digital Designer and often have trouble when I accidentally use bricks "without bottom tubes". Another thing that trips me up are tiles of "undetermined type".

I was having strange things happen when I added bricks that were the same as bricks already present in the list. Instead of collecting the two rows together and summing the quantities they seem to stay as two separate records and strange things would happen. I'm also a bit weary of setting quantities to zero.

Another thing you should be able to do, rather than deleting items, is to double click the 'green tick' mark in the first column a couple of times such that it becomes a 'red cross'. I haven't been able to workout what the 'blue plus' state means but setting it the the cross seems to have the same result as deleting it all together and makes it easier to keep track of what you are doing with your list.

[tsrplatelayer]

Yes - double clicking the tick makes it 'exclude' but I found it unreliable without saving, closing BF and restarting.

I think the following would be useful to have.

1. A user option to select "Ignore if fewer than 'x' stores have stock" and allow drop down selection of 'x' with a default of, say 25.

2. A list if ignored parts added to he report with the reason it was ignored, i.e. too few stores.

3. Ignore parts that have no price info available, rather than aborting.

Ignoring parts could be achived by changing status from include to exclude during the price scanand then then running through the ignored list and setting the status back to include at the end of the processing.

Allan

[S_Bartfast]

Yeah, there are a number of little tweaks like that I would like to have.

I've even tried to contact the creator a couple of times and have offered to add them myself but haven't heard anything back.

Hopefully in time such things will be added because it really is a great little package and a few simple tweaks like that would make things a lot nicer to work with

[Calabar]

1. ... and exclude those with no price available.

2. ... exclude scarce parts

I thought that brickficiency already excludes unavailable parts with a preliminary check and can deal with scarce parts.

Am I wrong?

I've even tried to contact the creator a couple of times and have offered to add them myself but haven't heard anything back.

That's strange, mdoupe is an active user.

Maybe he have few free time for this project, but he should answer to private messages, at least.

mdoupe, are you there?

[mdoupe]

mdoupe, are you there?

Yes, I just did not want to give away the code at the time.

I may be ready to open source it now though.

[S_Bartfast]

Yes, I just did not want to give away the code at the time.

I may be ready to open source it now though.

Cool, if you do, please do let me know.

BTW I had also tried to contact you through your Google+ account but it seems you don't use that account much anymore.

This is what I'd said:

Hello Myles.

I just came across your Brickficiency software and must say I am most impressed.

The reason I'm contacting you is that I would like to investigate as to whether it would be possible to add shipping costs to the algorithm.

I live in Australia and I used your software to organise a 600 piece order recently and while it was able to find a very good combination of BrickLink stores according to their listed prices the shipping charges ended up being more than the price of the parts themselves.

In particular I ordered 15 EUR from Brick-Fan and was charged 24.30 EUR in shipping.

I mean no disrespect to Brick-Fan in particular but I would like to have been more aware of what the final cost was going to be and to potentially have the optimisation account for it.

I see that while another program called DLL Manager does not do any optimization it does know the weight of the bricks.

I'm wondering if it turns out that adding shipping information to Brickficiency is infeasible might it be possible for Brickficiency to at least display the total mass of an order so as to enable the user to determine the total cost by hand.

Anyway, I'm a programmer myself and I was thinking that if you're too busy to look into this I'd me more than happy to look into adding such functionality myself.

All the best,

Slarti.

In addition to the weight details I'd also like to add a few little tweaks, such as adding increment/decrement buttons to the 'Quantity' field and adding an "Either" option for condition (as in to search for both new OR used bricks).

Obviously you'd still be in charge of the program but I'd like to be able to submit such changes for you to verify which you could then either include or not.

Edited July 7, 2015 by S_Bartfast

[tsrplatelayer]

I thought that brickficiency already excludes unavailable parts with a preliminary check and can deal with scarce parts.

Yes it does handle scarce parts - in so far as the log lists the number of stores with stock. This log can be used to 'exclue' scarce' parts to have a better chance of a solution.

This process could be avoided with a user selectable minimum no. of stores. I would offer to have a go myself but my programming skills are very much at the beginner level.

Another 'nice to have' would be a 'best incomplete solution' i.e a solution for 2 (3,4 etc) stores with the lowest number of 'leftovers'.

As willl be seen from my earlier posts where I listed the extracts from the log it does not deal well with parts that have an empty price page.

Getting Price Guide information for Black Crane Arm Center

Retrying...

Retrying...

Retrying...

Retrying...

Error downloading price guide (got page, but there was nothing on it)

Some items were unavailable. Calculation cancelled.

In fact it wasn't the crane arm that cused the problem, it was a completely different part. This and coupled with re-running giving different errors was frustrating but now I have a way of using iy that works for me I am happy.

Despite these issues it is a great piece of much needed software, the author is to be congratulated

If anyone wants the BSX file I was using I would be happy to send it.

[S_Bartfast]

Yes it does handle scarce parts - in so far as the log lists the number of stores with stock. This log can be used to 'exclue' scarce' parts to have a better chance of a solution.

This process could be avoided with a user selectable minimum no. of stores. I would offer to have a go myself but my programming skills are very much at the beginner level.

Another 'nice to have' would be a 'best incomplete solution' i.e a solution for 2 (3,4 etc) stores with the lowest number of 'leftovers'.

Yeah, I have spent a lot of time thinking about these issues.

Part of the problem with showing 'best effort' results (as in fewest leftovers) is that we then start to have multiple objectives and multi objective optimisation quickly becomes an unwieldy beast!

As the program is currently it does several brute force optimisations, one with a single store constraint, one with a two store constraint, one with a three store constraint etc. In a way this program kind of achieves a multi objective optimisation by performing several single objective optimisations then ranking the results. If you start to add 'fitness' metrics that take 'near misses' into consideration this increases the search space immensely. As in, if you had say 100 distinct pieces in your list then there are 100 different ways you could exclude one piece (which you could possibly still bruit force) but there are 9900 different ways to exclude 2 pieces and almost one million ways to exclude three pieces, and each of those different lists would still need to have the entire optimisation run on it, and if you were able to do all that you would still have a problem of how to rank the results. Now admittedly you wouldn't necessarily have to brute force all possible combinations of pieces to leave out but never the less, if you start to consider 'near misses' things can get out of hand very quickly!

In terms of thinning out the list to give Brickficiency a better chance to find valid combinations of stores I would be inclined to leave this up to the user. If it were upto me I would have the data gathering step and the optimisation steps as two separate processes. As in, once the user has completed their list I would have the program interrogate the database to determine availability, and possibly pricing information, of each of the pieces and display this to the user before the optimisation is commenced. If any parts are unavailable or of very low availability the user could either exclude these from the list or make appropriate substitutions before starting the optimisation. This availability information should be relatively easy to include in the table and once obtained it should allow the user to sort their list by 'availability' and manually exclude all those parts that are unavailable (or have an availability of less than 25 if you wish ). If this could be done with a group select it would be easy enough for a user to do and I think it would essentially achieve what you're after.

I'm also not sure what setting the state of pieces to "Extra" (the 'blue cross') is meant to do but it would be nice if it meant they were not taken into consideration when performing the optimisation however when complete, if any of the stores happen to have any of the "Extra" pieces available they could be added to the order. It would effectively mean they were "Excluded" for purposes of the optimisation but in the final report they are noted as either being available from the given stores or not and could be added to the order accordingly.

[tsrplatelayer]

I'm also not sure what setting the state of pieces to "Extra" (the 'blue cross') is meant to do .......

I take your point abt the problems with 'leftovers'

For me the 'extra' blue cross only has meaning when downloading a set inventory into BS then it indicates the 'etxra' parts that are in the set box that are not required for the build when built in accordance with the BI. From a buyers perspective I only ever use the 'tick' or if I need to temporarily omit something from a search the 'cross'.

[mdoupe]

Yes, the extra status was included for compatibility with Brickstore Bsx files.

[mdoupe]

The time has come. I do not have nearly enough time to work on this project. I tried to deny it. I procrastinated because my code is ugly and horrible (I am not a programmer). With the recent changes to Bricklink, Brickficiency is broken. The most glaring is that logging in during calculation does not work anymore. I'm pretty sure it's just a problem with a regex, but once that is fixed, I imagine other things will be broken as well.

Because of this, Brickficiency is now open source (again).

https://github.com/BuildingOutLoud/Brickficiency

I am new to this. I have no idea how git works. But if someone is willing to show me the ropes, I would appreciate it.

[jolurove]

The time has come. I do not have nearly enough time to work on this project. I tried to deny it. I procrastinated because my code is ugly and horrible (I am not a programmer). With the recent changes to Bricklink, Brickficiency is broken. The most glaring is that logging in during calculation does not work anymore. I'm pretty sure it's just a problem with a regex, but once that is fixed, I imagine other things will be broken as well.

Because of this, Brickficiency is now open source (again).

https://github.com/B...d/Brickficiency

I am new to this. I have no idea how git works. But if someone is willing to show me the ropes, I would appreciate it.

that's a real shame. I only found out about this amazing piece of software a couple of weeks ago :( I hope it continues it's development one way or another.

[mdoupe]

There were people interested in developing it further. I just have to find them. :)

[tsrplatelayer]

This isnt my type of programming (I program control systems in assembler), but as a would be user I do hope you find a way forward. Good Luck.

One suggestion - perhaps the parts of the code that are related to BL website design/layout could be built into a sperate data file and loaded at run-time. This way any changes to BL will be easily incorparated, needing only a mod and re-release of the data file.

[S_Bartfast]

Brickficiency is now open source (again).

https://github.com/B...iciency

Woo-Hoo!

With the recent changes to Bricklink, Brickficiency is broken.

Orrr,

The most glaring is that logging in during calculation does not work anymore. I'm pretty sure it's just a problem with a regex, but once that is fixed, I imagine other things will be broken as well.

Hang on... it seems to still be working fine for me.

Is anyone else having this problem?

If so could I get you to post your BSX file?

Reading someone else's regular expressions is kind of like reading tea leaves (in that you just make up what you want it to say and hope for the best...) but given a hard night's hacking and a healthy dose of "single-malt" I'm fairly confident I'll be able to knock something out.

So anyone able to provide details of how to get Brickficiency to fail consistently?

The night is young and I feel the urge for some hacking (not to mention some single-malt )

Edited July 20, 2015 by S_Bartfast

[S_Bartfast]

One suggestion - perhaps the parts of the code that are related to BL website design/layout could be built into a sperate data file and loaded at run-time. This way any changes to BL will be easily incorparated, needing only a mod and re-release of the data file.

That would be possible but it probably won't be all that beneficial.

For instance this is the expression that gleans the price/availability information from the web page:

.*<TR ALIGN="RIGHT"><TD NOWRAP>.*?<A HREF="/store.asp\?sID=(\d*?)&.*?<IMG SRC="/images/box16(.).png".*?TITLE="Store: (.*?)" ALIGN="ABSMIDDLE">.*?</TD><TD>(\d*)</TD><TD.*? \D*([\d,]*)\.(\d+)$

This expression is used to extract 6 values from a line such as this:

<TR ALIGN="RIGHT"><TD NOWRAP> <A HREF="/store.asp?sID=15336&itemID=45854709"><IMG SRC="/images/box16Y.png" WIDTH="12" HEIGHT="12" BORDER="0" ALT="Store: Give Me A Brick" TITLE="Store: Give Me A Brick" ALIGN="ABSMIDDLE"></A> </TD><TD>1</TD><TD> ~AU $32.66

(The values it extracts are shown in red)

While it wouldn't be too difficult to store this expression in a user editable file the likelihood of a non-programer being able to maintain it should the format change is slim, and, well, if it's going to be maintained by a programmer then they might as well just edit the source.

[tsrplatelayer]

Just ran a test on a list of 14 freely available parts - worked for me.

Also on a list of scarcer parts and it correctly identified which parts were not available.

I didn't get the 'retrying' an 'empty page error' but that may be because the parts in BL for sale database havechanged.

I had not expected that a user would edit the 'config' file. I had thought it was easier to release an update 'patch' containing a new config text file rather then re-compile a new distro.

Also when (not If) BL makes change having all the URLs, file paths and regex expressions in one place means that they are easy to find and you (or anyone else doing the maintenance programming) can easily find them and none get overlooked.

But as a user, not the programmer I will just be happy if it keeps on working.

Good luck

[mdoupe]

Hang on... it seems to still be working fine for me.

In testing it to see if it worked on the new bricklink I inadvertently stumbled upon a different bug. The list has to have at least one item, but if that item has a 0 quantity, it will just hang. I took this to mean it was not working.

[S_Bartfast]

I had not expected that a user would edit the 'config' file. I had thought it was easier to release an update 'patch' containing a new config text file rather then re-compile a new distro.

Yeah, that definitely would be possible.

Things inside are a bit of a monolithic blob at the moment and trying to tease out the 'configuration' elements from the mass would be a bit of a challenge but it is certainly doable.

On the whole though it is all remarkably well conceived for someone who is not a programmer and I must say I am impressed with mdoupe's effort. :)

[S_Bartfast]

In testing it to see if it worked on the new bricklink I inadvertently stumbled upon a different bug. The list has to have at least one item, but if that item has a 0 quantity, it will just hang. I took this to mean it was not working.

Arr, yes. I had come across that issue before. I don't think that one's a result of any changes with BrickLink though as I'm pretty sure I encountered some pretty weird behavior around that before. I never quite worked out exactly what was going on but even after increasing the quantity to a non-zero value weird things still continued to happen and a restart was required.

At the moment I'm trying to workout a solution to prevent the table rearranging the rows mid operation. As in if you sort by "condition", select all the elements and change their "condition" value, only half of the rows seem to get modified. I'm pretty sure it's because the rows get rearranged mid operation and some of the rows get set twice while others miss out altogether. I'm not exactly sure what can be done about it but I'll get back to you when I have a solution.

Cheers,

Slarti.

[CombatBrick]

I can help out with development, if there is interest. Let's gather features/issues list. Might be a good idea to create a new development branch going forward.

Alex

[S_Bartfast]

At the moment I'm trying to workout a solution to prevent the table rearranging the rows mid operation. As in if you sort by "condition", select all the elements and change their "condition" value, only half of the rows seem to get modified. I'm pretty sure it's because the rows get rearranged mid operation and some of the rows get set twice while others miss out altogether. I'm not exactly sure what can be done about it but I'll get back to you when I have a solution.

Okay, I have a solution.

It's not what you would call an overly elegant solution but so far as I can tell it works and I have pushed it to the repository.

[Phoxtane]

If anyone's still interested in developing this further, I've got an idea: maybe we can sort parts by 'number of appearances' - that is, when you click on a part in the list, it gives you the sets it appeared in and in what quantity. If I could sort by this (from least to most number of instances or vice versa) it would help me decide what parts were rare and what weren't and how to split up my orders from there.

Maybe instead of this it could just be as simple as what Rebrickable uses to designate a 'rare' piece: If it appeared in that color in three sets or less. For example, 57906 in Dark Blue is considered to be rare* because it only appeared in that color in two sets.

(*This may not be a good example, as I was able to get nearly 100 of that piece in that color for seven cents a piece...)

In other words, I'd like some ability to see what may end up being the difficult pieces to get.

[S_Bartfast]

I have been working on something related to this.

Rather than counting sets and quantity and such it simply queries BrickLink and determines the availability by the number of stores that currently have that part in stock. As it is, when the report is generated, this number is display for each part at the top but I have been working on a way to fetch this information when pieces are added to the list rather than as part of the optimisation and displaying this information to the user in a "Stores Available" column. You can sort by this column in order to see the least available (i.e. the rarest pieces) and optionally exclude them from the optimisation if you wish.

I discussed the idea in an earlier post:

...once the user has completed their list I would have the program interrogate the database to determine availability, and possibly pricing information, of each of the pieces and display this to the user before the optimisation is commenced. If any parts are unavailable or of very low availability the user could either exclude these from the list or make appropriate substitutions before starting the optimisation.

Rather than lookup the availability of all the pieces in one batch however I have been working on looking it up when pieces are added to the list. I have the general concept working on my machine but it a bit flakey at the moment.

Does this approach satisfy what you are after?