Some contemplating on frame storage formats and clashes therein

I recently posted a little bit on how I now store contributed videotex (teletext and viewdata) frames within a database, so as to make accessing them far easier on the application side.

To do this, I had to decide on exactly how to store the visible content of the frame.  Everything else is easy; I crated a secondary table holding key=>value pairs, which means it is very easily expandable, and any application needing particular data can go look for it's own, and not be confused by anything extra.

So. The frame content itself.  I didn't get much help looking at existing storage formats, as I've got at least 17 types documented, and others I know about.  I may however have been influenced somewhat by them.

When you think about a viewdata frame, or a teletext page, you automatically see the 23-25 lines by 40 columns of static image.  Almost every frame you will find that has been saved out by a terminal emulator, or teletext captures, will consist of those 920, 960 or 1000 bytes of data, perhaps with some meta-data accompanying, sometimes not.  I think that every third-party viewdata host that I have so far encountered also stored its pages so.  Individual characters took up a single bytes as per their ASCII character code, and colour and control characters were also stored as a single byte.  For teletext, this uses the non-display codes below the space, as there is no concept of cursor movements, carriage returns, etc, on a teletext screen, which is what these values are used for in a serial-terminal based service.

Prestel, and viewdata generally, is however serial.  Frames are sent to the user as ASCII characters, but the colour and control codes are sent as command sequences:  Escape then a capital letter.  So, what might be stored in a teletext page as "<01>RED<02>GREEN<07>WHITE" would be sent to a viewdata terminal as "<ESC>ARED<ESC>BGREEN<ESC>GWHITE".  Short lines would be terminated by a carriage return and linefeed, so reducing the need to send the whole 40 characters.

Now.. Prestel itself is known to have stored the frame data exactly as it would be sent to the user.  There was a hard limit of 920 bytes available to the editor to use, and colour codes, etc, took up two of them.  This made creating complicated graphical pages somewhat difficult, as too many colour changes could quickly eat up all the allocation.  (Response frames were even worse; you only got 716 bytes to play with!)  This is probably why all third party viewdata servers stored their page as the 22x40 character full image, with the control codes stored as per teletext.  Doing this allowed for much more colour and graphic rich content than was possible on Prestel itself - the conversion was done on transmission.  The actual codes stored varied - some systems used 7 bit data throughout, some used top-bit sett letters to indicate that letter needed the escape sending before it, some used 7 bits for visible characters, and top-bit set control codes (codes in the range 128-159) and at least one had everything with the top bit set!

So fast forward 30 years, and I'm writing code to handle saved viewdata pages and display them on this new-fangled World Wide Web thing.  There is zero support for viewdata and teletext format images, so we have to roll our own, converting saved pages in any number of formats into PNG or GIF (to account for flashing characters) images that a web browser can display.

As an intermediate stage, I have to pull that 22-24x40 matrix of characters out, before plotting them onto a graphics image for sending to the viewer.  This intermediate block of characters I called an "internal" format, and was 7-bit clean, so codes below space for the colour codes, and the rest visible.

For nearly ten years this worked fine, and this internal, intermediate format, was the format used when I created the page database.

It is only this week I hit a problem with this, and it is down to a peculiarity with how Prestel stores Response Frames.  (And, I assume, other frames that are not simple static pages.)

A response frame contains a number of fields that are defined by the editor when they create it, and are either filled automatically by the Prestel server when it displays the page, or  can contain text or data to be entered by the user.  When the user hits # on the last field, they are given the option to send (or not) the page to the IP.  It is then delivered to their mailbox in a filled-in state.

When defining a response frame in the standard Prestel online editor, a field is specified by typing, e,g. Crtl-L n 30 Ctrl-L will create a field of 30 characters length containing the subscribers' name - on pressing the second Ctrl-L the system will display 30 "n"s in the required position.  The same procedure is repeated for any other field you request.  What gets stored in the Prestel database is a single Ctrl-L and 30 "n"s.

When you retrieve a page from Prestel using the "Bulk" Online editor, it is sent exactly as stored, so you get the Ctrl-L and sequence of letters alongside the Escape'd colour codes and CR/LFs for short lines.  Uploading a replacement frame you specify the layout in the same manner.

Those of you familiar with the standard ASCII control codes will recognise that Ctrl-L is also known as "Clear Screen", and is a character that is usually sent before sending the frame content.  This is probably why it was used for this purpose - finding it in the middle of the frame content would not make sense, so it was re-purposed as a flag for start-of-field.  Obviously this is never actually sent to the user, but is replaced by a space when viewing on a terminal.

Now ...

I have two small databases in my posession that were pulled back down from Prestel at some point, and these include a number of Response Frames.

When I converted the data to my "internal" format to load them into the database, this normalised the control codes to 7-bit data, filling that lower 32 bytes of the table.  On displaying, these codes were sent as <Esc><code + 64>, this recreating the colour sequences.

When it comes to a <ctrl l>, however, this was never stored in the database - the normalisation routine ignored it.  However, even if it had been saved, on recall, it would have been translated into an <Esc>L, the sequence to end double-height text.

So, to summarise, the normalisation I did, in most cases, lost the start-of-field character because it wasn't expected in a frame.  And if it did make it though, it would be indistinguishable from the "Single Height" code, and as that was allowed anywhere in a response frame, it couldn't be deduced from context.

I never noticed, because there were so few frames affected, and there was no need to process the fields the code indicated, anyway!

This last month, however, I've been working on a viewdata host program that will run on a modern server, and which I could use to receate the look and feel of using the original Prestel service.  I've been testing this using an actual Prestel terminal, and it's been great fun!  It's only when I stumbled across one of these response frames, and decided to support them, that I discovered this problem!

Looking into how other file formats solved this, it seems that at least one of them uses <Esc> itself as the field indicator.  If  stored in the database like that, when expanded on recall this would translate into an unused code sequence, in viewdata, so is a suitable alternative.  I will translate the affected pages, eventually!


So, a decision taken about 10 years ago came back to bite me this week. And it's all to do with 25 year old data in a file format determined 40 years ago that everyone else decided needed to be done differently.

Well done for making it this far!


As an aside .. Prestel added support for "Dynamic frames" which were basically frames that could contain cursor movement characters.  This meant you could go back and change things after you had already drawn them.  This was easy for them, as they stored data in an as-transmitted form anyway.  It's no so easy for host software that expects it's frames to be stored in a fixed matrix!  I'll be working on this, one I find some original examples....

The Videotex Database - submit your pages now!

When I started viewdata.org.uk (and teletext.org.uk), I just uploaded the pages and databases I had as-is, and had my scripts deal with them on an as-accessed basis.  This is because I wanted to preserve the data as much as possible - any translation to a new format (such as JPEGs) would inevitably lose data, as well as context.

As time has moved on, and as the variety of data formats I have had to deal with has proliferated, this has increasingly become somewhat unwieldy. I decided, therefore, to try and rationalise things somewhat.

Each of the various file formats I was dealing with had different properties. Each had strengths, and each had weaknesses.  I could not decide on a single common format to try and convert files into.

Rather than create a new "perfect" file format, I decided therefore to store the frames within a database.  By having a primary table for the page content and certain static data, and a separate table for meta-data, any particular properties a particular file format had could be accommodated.

Once the data is held within a standardised database, of course, it makes it much easier to access it and use it from many different applications.  The first, and most obvious, is the ability to search across the entire database for key words or phrases. This is implemented on the front page of the database.

The main in-browser viewer for the saved pages implements a timeline function, where you can see how a given page has changed over time.  See, for example, the CEEFAX news headlines.

And of course, for viewdata pages, once can implement a dial-up host, so 1980s terminals can connect directly into the service and browse it exactly as they did at the time.  (This is mostly done, just pending further tidying up!)

Currently the database contains page data I have collected myself or already been sent. However I am aware that there is a vast amount more out there.  Jason Robertson has been amazing at rescuing teletext pages off old video tapes, and I know of at least one previous Prestel IP that has a massive archive of pages still extant, albeit sat on very old hardware.  I've got part of The Gnome At Home, and I know the rest still exists.

This week's task (one of the various "I'll do something" for Retrochallenge 2018/09) was to create a page for viewers to directly submit their pages to the database.  This is now complete!  It actually places the data into a queue, after briefly validating it, so it can be checked and added later.  I would welcome any contributions, anything from a single frame to a complete service backup!  If you need help, feel free to drop me a line.

Retrochallenge Day 14

A little more work today.

The teletext viewer javascript currently expects to find the entire teletext service held within the html of the web page as encoded links.  This is great for a static archive, as it places very little load on the webserver.

I had created a quick bit of php that could construct the html page when loaded, which was the source of the pages I linked to last time.  This is all well and good, but does not help with the sort of interactive services I would like to use the viewdata version for.

So today's work has been adding functionality to request pages from a server, thus allowing for ever-changing pages to be served up instead.  This involved not only modifying the JavaScript, but writing a server in php to deliver the requested pages.   This also brought out some previously unnoticed limitations in my viewdataviewer class, which I was using to parse the stored data serverside.

So, the class has had some fixes added, server written, and viewer updated.  You can play with it here, although there isn't actually anything dynamic on it at the moment.  Subpages aren't quite working right yet, but most of the rest is!

(I did break the first demo, btw, in case you were reading yesterday's post today, when I fixed a "bug" in the class file, but hadn't removed the work-around in the html generator!  It's fixed, now!)

Retrochallenge day 13

Blimey, where are the days going...

OK. I've taken the hacked-about teletext-editor-that-acts-as-a-viewer, and split off the hacks.  Then I modified the (latest version of) the teletext editor so that it exposes a bit more of it's internals, as the viewer needs that access...

It took a bit if trial and error, but seems to work.  Code is up on github and a demo is (temporarilly) here. I added a touch of code to allow direct linking to specific pages while I was at it!

Now to look at doing the viewdata browser that I was supposed to be doing in the first place!

Retrochallenge Day 2.. er... 8 ..er .. 10

Blimey, has it been a week already?

OK.  I've not done much coding since last time, but I have been reading code and daydreaming planning out my next move.

Now Javascript is not my strongest language.  I can read it, and modify it, but actually writing new code is a bit of a challenge.  Part of the rationale behind this task was to get myself a bit more familiar with this hideously back to front language...

The teletext browser I used is that created by Adam Dawes based on Simon Rawles (et..al.) edit.tf editor, and grabbed from Jason's captures at uniquecodeanddata.co.uk.  The modifications are to add a pile of new functions, and truncate and or redirect others.

As the original editor has moved on somewhat since this was done, it seems logical that, if I want to do more mods to it, then i should base my code on the latest version.  If I can do it in such a way that I do not need to actually modify the editor, just call it, then that would be best.   In PHP I would, assuming it was a class, extend the class in a new file and override the relevant functions.  So... How to do this in Javascript...

I tried using prototypes ... but hit the problem that the editor is written with lots of private variables and functions, which the new functions in the viewer refer to.  Using the existing editor as the viewer's prototype doesn't work because it cannot access the private variables.  Drat.

<days pass>

After spending more time than I ever expected looking at javascript objects, inheritances, etc., I have decided not to commit myself to ever having to do anything major in this language!!

Sticking with Javascript, I think the best approach at this point would, after all, be to fork edit.tf and modify it to separate out the actual display part from the editor part, that way I can provide for a viewer, indeed, different viewers...  Might even be a mod Simon would like...

Sigh.  Bloody Javascript.

The other option would be to go back to my own viewdata viewer class, which runs serverside to create the images.  I understand this, but I was hoping not to have to do this, as it makes updating the "screen" with the page number being keyed dependant on the server, rather than being local.


So, ten days in, and all I've achieved is discovering that what I thought would be a simple task is much more complicated than I thought it would be.

Retrochallenge: Day 1

OK. First day, and I have to do something.. .whether I can keep this up is another matter....

I had a look at the code used for browsing Jason's teletext captures.   These use a modified version of the edit.tf teletext editor (the 'viewer'), driven by an html page consisting of a mass of links!  The viewer grabs all these, displays the first one, then accepts key-presses to get the next page number, as per a teletext page.  Plus it allows up/down arrow shenanigans to skip through.

Teletext, as you should know, shares the exact same display format as Viewdata, namely 24 (or 25) lines of 40 characters of primary colour text and simple block graphics.  As control codes take up a space on the line, this makes it harder than you might think to do multicoloured images..

So, I've got a pile of dumps of teletext pages over at www.teletext.org.uk so the obvious thing to do is use one of those, see if I can use the viewer just as it is.  That way I have a starting point, and can begin to understand the code and decide on the particular direction I want to go.

I've got a php class in working-but-incomplete state that allows me to manipulate viewdata and teletext pages.  A quick bit of code to load one of the teletext archives and then spit out each page as a link took me significantly less than an hour, and only 14 lines of new code!

So...  from this,  to this.    I think that's a positive step.

Today has, however, shown up a lot of features currently missing in vv.class that I need to add in, particularly to deal with the viewdata side of things.  That is partly what the whole point of what this was for, though: to get an idea of what I need to do next!

Home console for Total Television

Home Console for total television?

Our cover picture, this issue, demonstrates that home electronics of the future need not be stark and futuristic. It is the first actual physical representation of how television and sound of the future might well look in the sitting room of the relatively near future.

This two-part furniture-treatment in glowing rosewood, was presented for the first time ever by TV science pundit James Burke at a Radio Industries gathering in London on the eve of the start of his recent BBC series "Connections", in which he took an idiosyncratic look at the ways TV might be used in the future. The "home video and computer console" as it was described, has been produced by the Thorn Group under its Ferguson brand, to present how it thinks such a console to cover known and expected advances in home electronics might be developed.

The two modules, moving easily on castors, can be brought together as a unit of furniture, or separated for use individually in different parts of a room.

Lest this should be thought an idle flight of fancy, it should be recorded that a 180-page report from the USA has stated that what it calls IVTs (integrated video terminals) will emerge within the next four years. The report predicts that IVTs will be a billion-dollar industry within ten years. It even puts a price of $1400 (£700 or so) on such units in 1982. 

With typical transatlantic zest for new word pictures, the report (from International Resource Development Inc) says the expected strong consumer demand for interactive TV services will lead to the concept of "narrowcasting" as opposed to broadcasting.

The IVT is seen as being the primary home tool for: entertainment, publishing access, home environment scheduling, home admin (payment of bills etc), domestic appliance control, self education and correspondence via electronic mail services.

International Resource Development Inc put Japan ahead when it reported originally, in August last. But it said that the US had a "major opportunity for innovative breakthrough" for the second generation of IVTs.

The report sees the terminals selling through retail outlets initially, but as it "assumes a greater and greater role in the control of other appliances and as the centre of home life, education and correspondence as well as entertainment", it would become more like the motor car in marketing, with specialised dealer outlets.

It sees manufacture moving eventually away from the present home computer manufacturers, and possibly from the present TV makers too, to the "vertically integrated companies with solid-state computer and consumer manufacturing and sales experience"—companies like Texas Instruments and IBM are named as "likely candidates".

Key to the controls -

Starting from the left, the units contain: 

1 A display panel for digital time and  programme selection by LCD (liquid crystal display).

2 Drawer one, with removable remote control for TV programmes, teletext and viewdata operation. Also a
"vote "button by which, if cable TV is established, immediate response to questions asked in a TV studio can be accurately measured. This is already on test in the US.
3 Drawer two, a home video system — the all-electronic operation reduces the size from today's bulkier models.
4 Drawer three, has a floppy disc unit — computer memory store for instant information retrieval.
5 Drawer four, a stereo audio cassette unit with all-electronic function controls. Cassettes can also store
viewdata or other pulse-coded information.
6 A telephone handset moulded to match the cabinet. To be replaced, when available, by a videophone instrument.
7 Drawer five, a keyboard for home computer programming, advanced video games and viewdata messages, plus a print-out facility for messages or processed information.
8 A retractable small screen monitor for checking incoming or outgoing information to be processed by other elements of the console.

And, of course, the large wall-screen with monitor screens for showing other programmes, viewdata and teletext messages, or even pictures from in-house closed-circuit cameras for security or domestic needs.

First published in Viewdata and TV user, January 1979.

(Just think; you could do all that today on a mobile phone! Maybe I should do an emulator..)

Where are they now?

Or... probably more accurately, "do they know who they were?"

As part of my work on viewdata.org.uk I've also come across, and been sent, quite a few pages saved from the various teletext services about at the time. I've started putting these up on teletext.org.uk which, as yet, is really more of a dumping ground than anything informative.

Anyway, when looking through these things, one finds things like this:

This dates from about 1986. Guessing that the kids who contributed would have been about the 10yo mark, that puts them at about 34 now, give or take. I wonder if Maria, Marie, Mariam or Sian knew that their words would still be remembered a quarter of a century later.