Rob's Ramblings

Sunday, 29 March 2015

Minecraft, Spigot, Bungeecord and Docker!

Tiddler has lately got really "into" Minecraft, spending rather too much time watching YouTube videos (Thanks, Stampy!) and creating her own worlds..

From my point of view, I quite like the idea of being able to add custom mods, and the (mostly) friendly and open ecosystem that has developed around it.  It reminds me very much of my time spent in, and writing patches for, WorldsAway, oh god, nearly 20 years ago... That was a fun time... it's a pity they didn't operate a bug bounty - I found quite a few!

Anyway, back to Minecraft.  She wanted to play with me, so we started off by my downloading and running the official "vanilla" server on one of the linux boxes.  It works, but is limited.  So I go find out that there is lots of alternate server software available. We end up with a two or three on different ports.  But the little box I was running it on really couldn't cope - a low end pentium that usually just ran an asterisk server!

So lets upgrade.  An impulse eBay purchase a few months back had bought me a Dell Poweredge 1950.  A rack server, which was currently sitting idle after an initial play.  With Dual 4-core Xenons and plenty of RAM, it was obsolete, but still probably the most powerful machine in the house! It was just itching to be used.

Now, with the prospect of letting the public into the minecraft server at some point, and with an urge to "do things properly", I wanted some form of isolation for the server processes.  Running a full blown VPS or virtual machine felt a little overkill, so I started looking around at what was recommended.  Now, years back, I used to use jails on FreeBSD, and it seems that the current flavour of the month builds on that concept, and is called Docker.

Now Docker does seem like the bees knees.  Reasonable isolation of a process with very little overhead, and in an ostensibly portable fashion  Let's use that then.

Several days of playing and one steep climb up the learning curve later and I'm pretty familiar with the basics.

Now... Minecraft. ... How to do multiple servers or worlds easily. This brought me to Bungeecord.  It is basically a proxy, that will control and switch users between servers.  That's better - no need to set up an entry on each client for each server - just do the one, and we can change servers on-the-fly once logged in, and I control all the config from the server itself.  This seems to be closely tied to the Spigot server sofware, so we might as well use that for the servers.

OK. The configuration.....

Firstly, we'll use data containers for all the, well, data.  This means I can delete and re-run things without having to re-configure the servers all the time!

We're going to be supporting multiple worlds, so in this example, we will call them simply world0, world1, world2, etc.  You may wish to use more descriptive names.

Create a data container..

sudo docker run -t -i --name world0-data -v /minecraft busybox true

Create a server..

sudo docker run -t -i --volumes-from world0-data -e EULA=true --name world0-server  nimmis/spigot

That should all fire up and eventually leave you at a > prompt - type stop to exit.

Now the more discerning among you will have noticed that we didn't specify any ports to reveal on that run command, so nothing will be able to connect to the server!  I know; that's deliberate.  We don't want just anybody connecting, after all.

Next, we need to adjust the config on the server, because bungeecord requires the individual servers to be in offline mode - it handles the authentication itself.   The config will be in /minecraft/, which is within the data container, so it's easy to create temporary access

sudo docker run --rm -t -i --volumes-from world0-data centos /bin/bash

cd /minecraft


You can also use the same technique to install any mods, ready-made maps, or other customisations.  If you need to install any extra programs into the container to allow you to fetch these, e.g. wget, ftp, scp, etc., then that's fine - you are working in a temporary container, and you will not affect the programme container, keeping it lean.  Only items you place into /minecraft - the data container - will be preserved!

Repeat for as many worlds/servers as you wish to create!

Now to use bungeecord to link them all together,

Create a data container for that too..

sudo docker run -ti --name bungee-data -v /bungeecord busybox true

And then create the server container..

sudo docker run -ti -p 25565:25565 --volumes-from bungee-data -e EULA=true \
  --name bungee --link world0-server:world0 \
  --link world1-server:world1 --link world2-server:world2 \

This time we are exposing a port to the world!

Make sure you add a --link for each server you created and wish to give access to.

After it fires up, and finishes creating everything, key "stop" to exit.  The rehf27/bungeecord image drops to a bash prompt on exit, so you can edit the config directly here.

vi /bungeecord/config.yml

If you need to do anything more exiting, or adjust things later, you can use the same technique as before to access the configuration from a temporary container -

sudo docker run --rm -t -i --volumes-from bungee-data centos /bin/bash

So, edit Bungeecord's config file, and change as appropriate.  The main things to alter are the listener ports - change to 25565 (as per the run command above) and the list of servers:

    motd: '&1Lobby Server'
    address: world0:25565
    restricted: false
    motd: '&1A New World'
    address: world1:25565
    restricted: false
    motd: '&1Another World'
    address: world2:25565
    restricted: false

- the "world0", "world1", etc in the address field comes from the alias, the part after the colon in the --link options on the run command.  Each of these is placed in the /etc/hosts file on this container, so that you can locate them, and specifying the link grants the access, so you don't need to open ports!

Save, and exit.

Now. start all your servers if they are not already

sudo docker start world0-server
sudo docker start world1-server
sudo docker start world2-server
sudo docker start bungee

Then point your minecraft client at your server IP, and have fun!

If you add or remove worlds later, you will need to crate a new bungeecord container with the new links.  Delete the current one..

sudo docker rm bungee

and create the new one, eg..

sudo docker run -ti -p 25565:25565 --volumes-from bungee-data -e EULA=true \
  --name bungee --link world0-server:world0 \
  --link world2-server:world2 \
  --link world3-server:world3 --link world4-server:world4 \

Because we used a data container, all existing configuration, mods, plugins, etc., will be preserved!  You can open a temporary container to edit the config file for the changes to the list of hosts just as we did before.

One caveat - stopping and starting a server will give it a different private IP address, which means the bungeecord server won't be able to connect to it - you will have to restart the bungee server to have it update things.  (There may be an option somewhere to preserve the private IP assigned to a container, but I've not yet looked for it.  If there is, use it!)

This took me several days of fun and games to tie down; I hope it's useful to others.  

A quick PS - I've not gone into depth on how to configure Spigot and Bungeecord;  just the essentials relating to their interaction within docker.  There are many guides and help resources out there that will assist you there.

For those of you that outgrow a single physical server, having your individual services Dockerised (is that a word yet?) means you can easily transfer them to another server with virtually no changes!  This useful link shows how to configure "one" docker across multiple physical hosts!

There also seems to have been work to add new --links to running containers; with a little bit of extra code to  interrogate the environment, this should make it possible to dynamically create the Bungeecord config file (or the hosts list at least) and avoid some of the additional configuration steps necessary to add new servers.

There are lots of ways to improve all this, but the aim of this write-up was to show how I got things working in the first place! :-)

Tuesday, 6 January 2015

How to annoy your users.

1. Use percentages to decide if a seller is "top rated" or "below standard".  Never mind that fact that a high volume seller can create 100 unhappy customers without even dropping a feedback percentage point, and keep their "top rated" status, but just a couple of mistakes by someone who manages to sell a few dozen times a month means they are damned for all eternity.  Where does "protecting the customers" feature in that?  High volume = piss off as many people as you like; we don't care.

Particularly annoying since the mistakes were caused by a third party (official!!!!!) software we used over-selling items. I could thus justify a "Not Our Fault" as it's not easy to keep track of over 1000 items manually, which was why we wanted to automate it!

2. Stop that small seller from listing any more items, so they don't have a hope in hell of raising their performance levels again whilst still charging them £60 per month for the shop they now can't use.

3. Restrict the seller's own personal account as well, because they are "linked" even though they are plainly used for totally different purposes.

4. When the seller attempts to contact you about the personal account, reply back telling them that their user-id violates their user-id policy, despite it having been used unchanged for nearly 15 YEARS, and if  they don't change it immediately, it will be changed for them, and cannot then be changed again for a further month.

I think that after 15 years, it is at the very least an implied contract term that they find the name acceptable.   Particularly as we have been in contact several times before this incident over unrelated matters!

5. Refuse to discuss even the possibility of compromise.   Refuse to even close the accounts when asked to do so.  Because they can't close an account that is restricted until it's performance is raised. Which it can't do because WE CAN'T SELL ANYTHING!  aaarrrrrrrrrrgggggggghhhhhhhhhh!!!!

I'll have to add up the figures, but I think the business actually LOST money trying to sell on eBay recently.  Sales have been low, with not enough gross receipts to even cover their fees, never mind cover the cost of the goods sold and make a profit out of it.  

I want to swear very loudly and break something.  This post will have to do for now.

Labels: , ,

Sunday, 3 November 2013

Home console for Total Television

Home Console for total television?

Our cover picture, this issue, demonstrates that home electronics of the future need not be stark and futuristic. It is the first actual physical representation of how television and sound of the future might well look in the sitting room of the relatively near future.

This two-part furniture-treatment in glowing rosewood, was presented for the first time ever by TV science pundit James Burke at a Radio Industries gathering in London on the eve of the start of his recent BBC series "Connections", in which he took an idiosyncratic look at the ways TV might be used in the future. The "home video and computer console" as it was described, has been produced by the Thorn Group under its Ferguson brand, to present how it thinks such a console to cover known and expected advances in home electronics might be developed.

The two modules, moving easily on castors, can be brought together as a unit of furniture, or separated for use individually in different parts of a room.

Lest this should be thought an idle flight of fancy, it should be recorded that a 180-page report from the USA has stated that what it calls IVTs (integrated video terminals) will emerge within the next four years. The report predicts that IVTs will be a billion-dollar industry within ten years. It even puts a price of $1400 (£700 or so) on such units in 1982. 

With typical transatlantic zest for new word pictures, the report (from International Resource Development Inc) says the expected strong consumer demand for interactive TV services will lead to the concept of "narrowcasting" as opposed to broadcasting.

The IVT is seen as being the primary home tool for: entertainment, publishing access, home environment scheduling, home admin (payment of bills etc), domestic appliance control, self education and correspondence via electronic mail services.

International Resource Development Inc put Japan ahead when it reported originally, in August last. But it said that the US had a "major opportunity for innovative breakthrough" for the second generation of IVTs.

The report sees the terminals selling through retail outlets initially, but as it "assumes a greater and greater role in the control of other appliances and as the centre of home life, education and correspondence as well as entertainment", it would become more like the motor car in marketing, with specialised dealer outlets.

It sees manufacture moving eventually away from the present home computer manufacturers, and possibly from the present TV makers too, to the "vertically integrated companies with solid-state computer and consumer manufacturing and sales experience"—companies like Texas Instruments and IBM are named as "likely candidates".

Key to the controls -

Starting from the left, the units contain: 

1 A display panel for digital time and  programme selection by LCD (liquid crystal display).
2 Drawer one, with removable remote control for TV programmes, teletext and viewdata operation. Also a
"vote "button by which, if cable TV is established, immediate response to questions asked in a TV studio can be accurately measured. This is already on test in the US.
3 Drawer two, a home video system — the all-electronic operation reduces the size from today's bulkier models.
4 Drawer three, has a floppy disc unit — computer memory store for instant information retrieval.
5 Drawer four, a stereo audio cassette unit with all-electronic function controls. Cassettes can also store
viewdata or other pulse-coded information.
6 A telephone handset moulded to match the cabinet. To be replaced, when available, by a videophone instrument.
7 Drawer five, a keyboard for home computer programming, advanced video games and viewdata messages, plus a print-out facility for messages or processed information.
8 A retractable small screen monitor for checking incoming or outgoing information to be processed by other elements of the console.

And, of course, the large wall-screen with monitor screens for showing other programmes, viewdata and teletext messages, or even pictures from in-house closed-circuit cameras for security or domestic needs.

First published in Viewdata and TV user, January 1979.

(Just think; you could do all that today on a mobile phone! Maybe I should do an emulator..)

Labels: , , , ,

Saturday, 26 October 2013

Prestashop - deleting redundant images.

One of the things I wrote when I first set up a webstore using Prestashop was a product import routine from one of our suppliers, that created the initial products.

Among other things, this included all the images for the product.

Now, the supplier provided separate thumbnail and large images in their links, which were dutifully imported. So each product ended up with two apparently identical images, but the second was bigger than the first! I'd not really bothered to clean this up, as it didn't really affect anything.

Now however, with eBay's new picture standards in the offing, and the standard prestashop eBay module being somewhat.. um... inflexible... (more on my work on this at a later date!) I've realised that I need to get rid of all these unnecessary images.

The end result is the rather simple little module below.  This will appear in the "Other Modules" category.  Once installed, and configure pressed, it will present you with a load of images!

Where a product has two (or more) images, numerically in sequence (so they were created at the same time) and the first is the cover picture, it will show the first two.  Alongside each pair is a checkbox, already ticked.

Scroll down .. If the pictures are the same, leave it ticked. If they are different, clear the tick.

If you press the submit at the bottom, it will delete the first image from the product for each line that you leave ticked.

This does for me.  It may not do for you.  Put your store into maintenance mode and make backups of your filesystem AND database before you even install this.  I offer this with no warranty whatsoever.  Do not complain if it all goes horribly wrong - just restore and then work out why.

Download ImgClean 1.0 here.

Labels: ,

Friday, 9 August 2013

Prestashop - switching to shared stock.

Another useful tid-bit of information.

If you use Prestashop for your online e-commerce, then you may be aware of their "Multistore" feature - basically, you can use the same installation to run completely different shops; the difference being the URL you access them from.

So what, you might think. That's not really much more use thant just running two copies on different virtual hosts. Well, the strength is that you can share products, customers, stock levels, discounts, pretty much everything ... if you want to. Plus, of course, it's only one installation to keep up to date.

Great, this suited an installation I did a while back. Unfortunately, although the products were set as shared across all stores, somehow in the several iterations of testing the option to share stock levels was not. Since the vast majority of products are unique to each store, with only a few cross-overs, nobody noticed, until now.

OK, simple, we thought.  Switch the option on now.

Share available quantities between shops of this group. When changing this option, all available products quantities will be reset to 0.

Eek... all quantities will be reset to zero....

Not what we want at all.

Checking the code, we see that yes indeed, when changing this setting, sure enough, it deletes everything from the "stock_available" table for the affected shops.

The stock_available table has two relevant fields in it - id_shop and id_shop_group.

When stock is held for an individual shop, the first is filled in, and the group is zero.  When it's shared between a group of shops, it's the other way around.

Since the only table affected when you do the switch is the stock_available one, it seems logical therefore that this is the only source of data that we need to preserve.

Much testing in my sandbox shop therefore resulted in the following bit of code necessary to preserve the stock across the change.  READ ON BEFORE TRYING THIS OUT!

update ps_stock_available set id_shop_group=1, id_shop = 0;

Yep!  All we need to do is update the group field, and clear the shop field!  Of course, if you have more shops that you don't want affected, you should include a "where" clause to restrict whose products this affects.

You need to do this before you change the shared stock setting.  Also, it will fail if you have any products with stock held individually.  This code will list any of these -

select p.id_product, pa.id_product_attribute, count(sa.id_shop) as shops
from ps_product p LEFT JOIN ps_product_attribute pa ON (pa.id_product = 
p.id_product ) JOIN ps_stock_available sa on p.id_product = sa.id_product 
and sa.id_product_attribute = IF(pa.id_product_attribute IS NULL,0,
pa.id_product_attribute) group by sa.id_product, sa.id_product_attribute  
having count(*) > 1;

Yep, that's a bit longer, sorry!  It's got three tables to cross-reference.

It it comes up with any, this code will tell you more about which stores an item is held in, and how much is in each store, etc -

select * from ps_stock_available where id_product = 782;

(That's for product #782, of course - adjust as necessary!)

If, as I found, the products had zero stock in the secondary stores, you can just delete all but one of the records -

delete from ps_stock_available where id_product = 782 and id_shop <> 2;

Re-run the check again, then do the conversion.

This is easiest, of course, from the mysql command line, but if you use a web-based tool, you should be able to manage if it will accept typed mysql command lines.

I should give you the obvious warnings -

BACKUP YOUR DATABASE FIRST!!  Preferably using outside tools such as mysqldump
Then examine the backup file - does it look like it holds all your data??
PUT YOUR SHOP INTO MAINTENANCE MODE. You don't want customers confusing things.
TEST IT THOROUGHLY. If you are not convinced everything is right, restore from your backup!

Labels: , ,

Tuesday, 16 July 2013

Retrochallenge 2013

In a fit of madness, I decided to re-enter this years competition.

If you cast your mind back, I entered last time with the intention of sorting out my "man cave" as the attic was described, in order to get the viewdata BBS back up and running.

Unfortunately, building work interrupted matters, and I didn't get very far.   More unfortunately, the building work stalled, and I still don't have a safe floor up there, so pretty much everything is still in boxes.  A year on.  The tale of what happened and what went wrong is going to make a great story at some point, but not yet.

So, this year, I've been hoping to get something working under emulation.  The ideal would be an emulated modem that can answer VoIP phone calls, and pass the user to an emulated server of some sort.  Modems and VoIP are reckoned not to work very well, but that's the "modern" types. V23 has no problem at all, as I proved when using the real stuff over it.

The trick is going to get a software modem working at such a slow speed!  IAXmodem uses SpanDSP, which proports to support the old FSK modems, including V23, but I've not yet managed to get it working.  Part of the issue is that the whole thing is geared up for faxing, so assumes some level of intelligence from the far end.  V23 modems, particularly the 25 year old ones, don't do anything intelligent!

So, that's the task this month.  Getting iaxmodem (initially) to answer a call from an ancient terminal.  Then I'll hook it up to a BBC micro emulator and run the BBS entirely virtually.

I'm acutely concious that over half the month is over, and I've not done anything yet!

Long term goal is to find a backup of the original Prestel software and databases, run up a GEC mini-computer emulator, and actually run the real thing.  But so far that's been elusive.  Anybody know of any surviving discs or tapes from Prestel itself?

Labels: , , ,

Monday, 18 February 2013


Please note:

If you wish to attach a length of timber to a brick wall, and you find that the screw stops going in any further, it's likely that you are doing something wrong -

This is to be expected if you didn't drill and plug the wall.  The screw will work it's way through the plaster, and stop when it hits the bricks.

This also, of course, means that it's not really attached to the wall, either.

Labels: , , ,