Templot Club Archive 2007-2020                             

topic: 3529File formats for T3/MEC
author remove search highlighting
 
posted: 29 Oct 2019 00:26

from:

Rob Manchester
 
Manchester - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Hi Martin,

Being able to export a folders worth of T2 files ( well those that are selected ) for MEC/OT will certainly be good going forward. Will you still see the warning message regarding the template labels when opening .box files created some time ago ?

Rob


posted: 29 Oct 2019 00:41

from:

Martin Wynne
 
West Of The Severn - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Rob Manchester wrote:
Hi Martin,

Being able to export a folders worth of T2 files ( well those that are selected ) for MEC/OT will certainly be good going forward. Will you still see the warning message regarding the template labels when opening .box files created some time ago ?
Hi Rob,

You can select all of them by ticking the export all box.

I think that message can probably be removed entirely now. Thanks for the nudge. :)

cheers,

Martin.

posted: 29 Oct 2019 06:39

from:

Graeme
 
Bangkok - Thailand

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
*I* wrote:
... while I waited for OTBOX files.
hahahaha What a ninny!

I just realised that I can create my own OTBOX files using OT!  Duh! :roll:
 
graeme

posted: 29 Oct 2019 12:32

from:

Martin Wynne
 
West Of The Severn - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Graeme wrote:
I just realised that I can create my own OTBOX files using OT!  Duh! :roll:
Hi Graeme,

Yes, the open-source versions are fully functional and usable in their own right. That was the whole point of releasing them.

The problem has been (apart from the missing functions) that their native OTBOX files are not cross-compatible with T2 BOX files. That's because they are both in binary format and Lazarus does not support the 80-bit extended floating-point format which is used throughout T2.

Which meant that users of OT would have no access to their backlog of T2 BOX files. When Rob mentioned that he was actually using TemplotMEC in anger, it dawned on me that doing something about that was quite urgent. Paul Boyd also mentioned that he was waiting for a solution.

I have now created a file transfer format provisionally called MECBOX which can be used to import and export files to and fro between OT and T2.

I shall shortly be posting scruff releases of both which users of T2 can use to create an archive of MECBOX files from their old BOX files. These can then be imported into OT as and when needed, and then saved again in OTBOX format for normal use.

There is also a new option in T2 (on by default), to create a matching MECBOX file whenever a BOX file is saved. So users of T2 won't need to think about doing that for their current work.

Ideally it would have been better to create an OT function to import BOX files directly. I tried to do that originally, but reading the 80-bit values in Lazarus got messy and error-prone. At some stage someone might want to have another go at that.

Changing the subject, the file viewer (and lots of other places in Templot, such as the save program preferences dialog) uses David Baldwin's THtmlViewer component. I originally purchased a licence for that, but it is now open-source and I believe it is included in some later versions of Delphi. But like everything else in Templot it is old, old, and you have probably found it difficult to find any Help files for it. That's because they are in the old .hlp format which is no longer supported in Windows.

However, I have some software which can convert old .hlp files to web pages (after a fashion), so I have done that for you at:
 
 http://85a.co.uk/templot_mec/thtml_viewer_help/

It's very scruffy, but usable. The font-size is very small and needs zooming for eyes which are even older.  :)

cheers,

Martin.

posted: 29 Oct 2019 15:15

from:

Graeme
 
Bangkok - Thailand

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:
Yes, the open-source versions are fully functional and usable in their own right. That was the whole point of releasing them.
OK OK - Don't rub it in - hahahaha


Ideally it would have been better to create an OT function to import BOX files directly. I tried to do that originally, but reading the 80-bit values in Lazarus got messy and error-prone. At some stage someone might want to have another go at that.
Absolutely! At some stage in the future we will be soundly cursed by someone who stumbles across an Alladin's cave of creations in T2 format and can't read them.

This is exactly the dopey kind of task I would eagerly offer to take on, but just now there is a scarily long list of more pressing matters. However, let's not lose sight of it.

BTW, a thought occurred to me.  We have a few formats out there now and that number will not, in my experience, go down. :D  Is it possible by reading the first few bytes of a file to determine what format it was written in?

However, I have some software which can convert old .hlp files to web pages (after a fashion), so I have done that for you ...
Thanks, I have not dug into this HTML stuff much, but this will no doubt be helpful in the future.

Cheers,

g







posted: 29 Oct 2019 20:51

from:

Martin Wynne
 
West Of The Severn - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Graeme wrote:
OK OK - Don't rub it in - hahahaha
BTW, a thought occurred to me.  We have a few formats out there now and that number will not, in my experience, go down. :D  Is it possible by reading the first few bytes of a file to determine what format it was written in?
Hi Graeme,

Not rubbing anything in. :)  See my recent remarks in the other topic:
This is a forum, not a personal conversation. I try always to remember that I'm writing for everyone to read, not only the person who asked a question. And that whatever I do write will be indexed on Google for 100 years and read by folks in 10 years time who are new to the whole thing. We are not trying to be Facebook, where today's post is tomorrow's chip paper.
If something appears on Templot Club which I think might be misleading to others, I try to clarify it, as much for them as in direct reply to the poster.


The Templot data files should be easily identified from their file extensions. If the same extensions are used by some other software, BOX and OTBOX binary files both start with the same string of 12 bytes:

03 4E 20 30 00 00 00 00 00 00 00 00

but BOX and OTBOX files can't be separated that way, they are both the same. Only the first 4 bytes are actually tested on loading.

There are probably very few OTBOX files in the wild, so it's not too late to change it, if it seems necessary, without preventing loading of existing files.

The new MECBOX files are text files, and at present start as:

DO NOT EDIT THIS FILE
=====================
template data - saved from Templot2 version 2.24.a
saved at 01:38:32 on 29/10/2019
29 templates from Downingham EM

which is invariant as far as "from ". But could be changed to anything we want.

cheers,

Martin.

posted: 29 Oct 2019 23:36

from:

Martin Wynne
 
West Of The Severn - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:
BOX and OTBOX binary files both start with the same string of 12 bytes:

03 4E 20 30 00 00 00 00 00 00 00 00

but BOX and OTBOX files can't be separated that way, they are both the same. Only the first 4 bytes are actually tested on loading.
Hi Graeme,

I have now changed the 12th byte to FF in OTBOX files to differentiate them from BOX files:

03 4E 20 30 00 00 00 00 00 00 00 FF

This is solely to identify the source of the file, it is not read on normal loading so doesn't prevent any existing OTBOX files from loading.

This will be in the scruff release of TemplotMEC which I'm planning to make shortly.

If you are planning to post some code for the file viewer soon, my condition will be fulfilled and we need to start talking about making the OT program download available on the Templot web site. That will require an installer, which presumably can be the same as the T2 installer? Also what happens about version numbers for T2 and OT/MEC -- should they advance in sync? Or start from scratch for OT/MEC?

cheers,

Martin.

posted: 29 Oct 2019 23:54

from:

Rob Manchester
 
Manchester - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Hi Martin,

Interesting to see the help you are receiving with development :)

I would have thought there needs to be some cohesion between T2 and MEC versions of Templot - assuming that any new features are to be rolled out at some point.

Pardon me for being a little confused here but if somebody were to write some code that implemented a new feature for MEC ( say outside slips or 90 degree Retford style crossings ) would that get added to T2 by you as well ? .....and are you to be a contributor to MEC and update the repository with your new chunks of code when in appears in T2 ?

Rob




posted: 30 Oct 2019 00:28

from:

Martin Wynne
 
West Of The Severn - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Rob Manchester wrote:
Pardon me for being a little confused here but if somebody were to write some code that implemented a new feature for MEC ( say outside slips or 90 degree Retford style crossings ) would that get added to T2 by you as well ? .....and are you to be a contributor to MEC and update the repository with your new chunks of code when in appears in T2 ?
Hi Rob,

You are not the only one a bit confused. :)

The first thing to establish is whether this latest interest in OT/MEC turns out to be lasting, or is merely a flash in the pan. We have been here before, remember. Time will tell.

If it does, then as far as getting everything currently in T2 into OT/MEC -- yes that's my intention. After which OT/MEC would become the "official" public version of Templot. And yes I would continue to contribute to it -- with others. But that happy state is going to take some time to achieve, and a perfect equivalence may not be possible. Some things may have to be let go, for example direct scanning into picture shapes.

In the meantime if someone posts some code to do say Dundalk Square Crossing in MEC, ideally I would want to incorporate it into T2. But it needs their permission to do so, because I can't just use other people's open-source stuff in closed-source T2 as of right.

What I don't want to happen is that I end up as the sole contributor to OT/MEC, with the extra work of open-sourcing everything. If that happens I would rather leave OT/MEC as-is and continue with T2.

cheers,

Martin.

posted: 30 Oct 2019 00:41

from:

Rob Manchester
 
Manchester - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:
Rob Manchester wrote:
Pardon me for being a little confused here but if somebody were to write some code that implemented a new feature for MEC ( say outside slips or 90 degree Retford style crossings ) would that get added to T2 by you as well ? .....and are you to be a contributor to MEC and update the repository with your new chunks of code when in appears in T2 ?
Hi Rob,

You are not the only one a bit confused. :)

The first thing to establish is whether this latest interest in OT/MEC turns out to be lasting, or is merely a flash in the pan. We have been here before, remember. Time will tell.

If it does, then as far as getting everything currently in T2 into OT/MEC -- yes that's my intention. After which OT/MEC would become the "official" public version of Templot. And yes I would continue to contribute to it -- with others. But that happy state is going to take some time to achieve, and a perfect equivalence may not be possible. Some things may have to be let go, for example direct scanning into picture shapes.

In the meantime if someone posts some code to do say Dundalk Square Crossing in MEC, ideally I would want to incorporate it into T2. But it needs their permission to do so, because I can't just use other people's open-source stuff in closed-source T2 as of right.

What I don't want to happen is that I end up as the sole contributor to OT/MEC, with the extra work of open-sourcing everything. If that happens I would rather leave OT/MEC as-is and continue with T2.

cheers,

Martin.
Hi Martin,

Yes, that makes sense from your point of view.

Would need a custom gauge setting for Dundalk -> EM(I)-SF for my odd flangeway size :roll: I will stick to Retford (style ) crossings in memory of Roy....

Rob

Last edited on 30 Oct 2019 00:42 by Rob Manchester
posted: 30 Oct 2019 01:30

from:

Graeme
 
Bangkok - Thailand

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:
I have now changed the 12th byte to FF in OTBOX files to differentiate them from BOX files:

Good move! :D   In my experience, relying on extensions is just too flaky and would come back and bite us one day.

Could you please let me know what you changed for this (maybe a couple of before/after clips) and I will put it into my version. I do not want to release anything which writes OTBOX files without this byte set.

This is solely to identify the source of the file, it is not read on normal loading so doesn't prevent any existing OTBOX files from loading.
I think a firmer stance on this is justified, but this should be fine, given that (hopefully) the OTBOX format will be short-lived and we will be able to use a text-based format shortly. :D

Cheers,

graeme


posted: 30 Oct 2019 02:17

from:

Graeme
 
Bangkok - Thailand

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:
The new MECBOX files are text files, and at present start as:

DO NOT EDIT THIS FILE
=====================
template data - saved from Templot2 version 2.24.a
saved at 01:38:32 on 29/10/2019
29 templates from Downingham EM

which is invariant as far as "from ". But could be changed to anything we want.
Hi Martin,

These are not 'obviously' comment lines (i.e. starting with '#' or ';' or any of the other myriad ways comments are denoted). in fact I am not clear which bits are comment and which bits data. What is the underlying file format you have used?

Given the BOX/OTBOX experiences, I would strongly suggest putting a file format version number in there too. (The Templot version can stay, of course, It is handy for fault tracking.)

Could you outline the file structure you intend to use?

Cheers,

g

PS
This is drifting off topic. May I suggest it is moved to a new topic "File Formats" or some such?
Last edited on 30 Oct 2019 02:43 by Graeme
posted: 30 Oct 2019 02:46

from:

Martin Wynne
 
West Of The Severn - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Graeme wrote:
Could you outline the file structure you intend to use?
Hi Graeme,

Here is a bit more of it:

DO NOT EDIT THIS FILE
=====================
template data - saved from TemplotMEC version 2.24.a
saved_at 03:11:43 on 29/10/2019
26 templates from Downingham EM
 
0000-i=26
 
```
0999-i=0
1000-s=N 0
1005-i=145686781
1010-s=01/08/2019
1015-s=02:33:51
1020-s= BH • EM  half-diamond K- 8  +  V- 8   RH
1025-s=Downingham EM
1030-s=[slip 023300]  slip half-diamond
1035-b=0
1040-i=0
1045-i=0
1050-i=0
1055-e=72
1060-e=0
1065-b=0

etc.

There are over 450 such settings for each template (depending on the number of shoved timbers), followed by the info and memo strings, and then it repeats from ``` for the next template:


3310-e=1650
3315-e=0
3320-e=73.0832727547584
3325-e=146.166545509517
3330-e=12
3335-b=0
3340-b=0
3345-b=0
 
`!`!`

template generated at 03:37:31 on 01/08/2019 using Templot v:2.23.c|scale = 4.0 mm/ft     scale ratio = 1:76.2|track gauge = 18.2    flangeway gap = 1.0|template: curved (constant radius)|rail head only (bullhead): rails vertical|------------|RH regular half-diamond, fixed K-crossings (fixed diamond):|1 in 8.00 RAM  ( 1 in 8.03 CLM ) regular V-crossing|1 in 8.00 RAM  ( 1 in 8.03 CLM ) K-crossing|equalized-constant timbering|------------|main-road centre-line radius = [ -1650.0] (constant radius)|diagonal-road centre-line radius = [ -1650.0] (regular half-diamond)|------------|adjacent track centres main side = 44.67|adjacent track centres turnout side = 60.67|angle at TXP crossover mid-point (CTRL-5) = 7.13 degrees ( 1 in 8.0 RAM )|angle at TVJP turnout road vee joint (CTRL-6) = 7.13 degrees ( 1 in 8.0 RAM )|------------|equivalent straight template dimensions BEFORE curving :||        overall length = 472.51|        exit track in  60 ft rails / 25 sleepers per length ( rail length = 240.0 ):|        exit track length = 268.83  ( 1 full rail lengths + 3 sleepers in 12.01 % of a rail length )||        diagonal-road centre-line radius = straight (regular half-diamond)|        V-crossing entry-straight (curve-end to fine-point) = 16.0||        half-diamond main-road inter-length (CTRL-0 to MCP) = 146.17|        half-diamond virtual lead (K-crossing to fine-point) = 146.73|        half-diamond actual lead (K-crossing to blunt nose) = 148.74||        knuckle bend radius (normal) = 32.0|        blunt nose to timber A = 1.33|        width of blunt nose = 0.25||        wing rail reach length (main-side) = 16.0|        wing rail reach length (turnout-side) = 16.0|        check rail overall length (main-side) = 52.0|        check rail overall length (turnout-side) = 52.0|------------|smallest radius on this template = 1650 mm ( 65.0 " )|total angular swing on this template = [ -16.41] degrees ( [ -1 in 3.4] RAM ) (in main road)|------------|nominal gauge :   EM       18.2  mm    4 mm/ft     1:76.2    EMGS dimensions         |------------|template location on trackpad :||rotation :  X = 0   Y = 24.0   K = 204.57 degrees ( 1 in 2.19 RAM )|   shift :  X = 951.38   Y = 1931.5|rail-end :  X = 951.38   Y = 1955.5||peg from origin :  X = 947.59   Y = 1963.78   K = 204.57 degrees ( 1 in 2.19 RAM )|peg from notch :  X = 947.59   Y = 1963.78   K = 204.57 degrees ( 1 in 2.19 RAM )||radial centre :  X = 1633.62   Y = 463.16||track centre-line radius at peg = [ -1650.0]  ( [ -64.96] " )|------------||total timbering length on this template = 2014.0
 
`@`@`

 your memo notes for this template ...|
 
`~`~`
 
```
0999-i=1
1000-s=N 1
1005-i=232470062
1010-s=01/08/2019
1015-s=02:34:28

etc.

I will write more soon.

cheers,

Martin.

posted: 30 Oct 2019 04:11

from:

Martin Wynne
 
West Of The Severn - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Graeme wrote:
Could you please let me know what you changed for this (maybe a couple of before/after clips) and I will put it into my version. I do not want to release anything which writes OTBOX files without this byte set.
Hi Graeme,

The editor formatting on here is a bit iffy*, so I have put the code changes in a web page:

 http://templot.com/companion/code_snips.php

cheers,

Martin.

*it was fine until the browsers wrecked it with their endless changes. There is no agreed HTML standard for editable text in design mode.

posted: 30 Oct 2019 09:01

from:

Graeme
 
Bangkok - Thailand

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:
The editor formatting on here is a bit iffy*, so I have put the code changes in a web page:

 http://templot.com/companion/code_snips.php
All done.  Thanks

posted: 30 Oct 2019 09:47

from:

Martin Wynne
 
West Of The Severn - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Graeme wrote:
PS
This is drifting off topic. May I suggest it is moved to a new topic "File Formats" or some such?
Hi Graeme,

Now done.

cheers,

Martin.

posted: 30 Oct 2019 11:23

from:

Graeme
 
Bangkok - Thailand

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:
PS
This is drifting off topic. May I suggest it is moved to a new topic "File Formats" or some such?
Now done.
Excellent!  Thanks.

posted: 30 Oct 2019 11:41

from:

Graeme
 
Bangkok - Thailand

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:
Here is a bit more of it:

DO NOT EDIT THIS FILE
=====================
template data - saved from TemplotMEC version 2.24.a
saved_at 03:11:43 on 29/10/2019
26 templates from Downingham EM
 
0000-i=26
 
```
0999-i=0
1000-s=N 0
1005-i=145686781
1010-s=01/08/2019
1015-s=02:33:51
1020-s= BH • EM  half-diamond K- 8  +  V- 8   RH
1025-s=Downingham EM
1030-s=[slip 023300]  slip half-diamond

... etc ...

Hi Martin,

Thanks for this.

This is not too far removed from the old 'ini' files from DOS days. Basically a simple store of key-value pairs (which is pretty much to be expected for this kind of application).

Field Names
We spoke earlier about readability and its importance. To help this I would strongly suggest having meaningful (and unique) names for the fields as it will make life easier when viewing these files.

For example, lines 1010 and 1015 are both called "s" but one is a date and one is a time.

If all 450 settings are unique items it may be a bit of a challenge dreaming up meaningful names for them all, but I suspect that this is not the case. I would imagine that there are 'groups' e.g. one set of values for each shoved timber, etc.

Structure
The data we are storing clearly has a structure (i.e a file is some header data followed by a group of templates; a template is a group of settings followed by some info strings followed by some memo strings; I would guess the settings also have some structure within them - for example the date and time on 1010/1015 are probably both related  to the same thing. However, the file sample has no structure to it - it is 'flat'.

I would suggest that we adopt one of the existing file formats which have been designed for storing and transmitting data such as this. This would have the benefit of familiarity to any other developers we entice to join the project and also give us tools that we can use to process the files, for example viewers which understand the structure and can present it nicely.


Although there are many such formats, I believe the requirements of representing structure and being readable narrows the list to relatively few. I have prepared a version of the example data in each of what I consider to be the main contenders.

TOML (Tom's Obvious Minimal Language)
The original INI file format fails to provide for structure. TOML is very similar, but addresses the structuring issue:

#DO NOT EDIT THIS FILE
#=====================
[overview]
format-version = 3
program = "TemplotMEC"
version = "2.24.a"
saved = 2019-10-29T03:11:43
template-count = 26
[templates]
  [templates.1] # Note indenting allowed but not required
  [templates.1.settings]
  index = 0
  something = "N 0"
  bignum = 145686781
  sometime = 2019-08-01T02:33:51
  somedesc = "BH • EM half-diamond K- 8 + V- 8 RH"
  name = "Downingham EM"
  type = "[slip 023300] slip half-diamond"
  ...etc ...
    [templates.1.infos]
      [templates.1.info.1]
        text = "memo text ... "
    [templates.1.memos]
      [templates.1.memos.1]
        text = "memo text ... "

  [templates.2]
    [templates.2.settings]
      index = 7
...etc ...

This is a simple file format, and reasonably readable, but not as well known or well supported as others.

XML

<!-- DO NOT EDIT THIS FILE -->
<!-- =================== -->
<templot-box>
 <overview>
  <format-version>3</format-version>
  <program>TemplotMEC</program>
  <version>2.24.a</version>
  <saved>2019-10-29T03:11:43</saved>
 </overview>
 <templates>
  <template>
   <settings>
    <index>0</index>
    <something>N 0</something>
    <bignum>145686781</bignum>
    <sometime>2019-08-01T02:33:51</sometime>
    <somedesc>BH • EM  half-diamond K- 8  +  V- 8   RH</somedesc>
    <name>Downingham EM</name>
    <type>[slip 023300]  slip half-diamond</type>
 ...etc ...
   </settings>
   <infos>
    <info>
     <text>info text ... </text>
    </info>
   </infos>
   <memos>
    <memo>
     <text>memo text ... </text>
    </memo>
   </memos>
  </template>

  <template>
   <settings>
    <index>7</index>
 ...etc ...
   </settings>
  </template>
</templates>

Yuck! Obviously, well known and well supported, but about as readable as a binary file.  :D  XML was never intended for this use, and it is a tragedy that it has become so pervasive for data serialisation.


JSON
Note that by design, comments are not permitted in JSON
{
  "comment-1": "#DO NOT EDIT THIS FILE",
  "comment-2": "#=====================",
  "overview": {
    "format-version": 3,
    "program": "TemplotMEC",
    "version": "2.24.a",
    "saved": 2019-10-29T03:11:43,
  },
  "templates": [
    {
      "settings": {
        "index": 0,
        "something": "N 0",
        "bignum": 145686781,
        "sometime": 2019-08-01T02:33:51,
        "somedesc": "BH • EM  half-diamond K- 8  +  V- 8   RH",
        "name": "Downingham EM",
        "type": "[slip 023300]  slip half-diamond"
   ...etc ...
      },
      "infos": [
        "info": {
          "text": info text ... ",
        },
      },
      "memos": [
        {
          "text": "memo text ... "
        }
      }
    },
    {
      "settings": {
        "index": 7,
    ...etc ...
    }
  }
}

Much more readable than XML, and pretty well supported (Lazarus libraries available) but still more verbose than necessary, and the clutter makes it harder to read.


YAML
---  #DO NOT EDIT THIS FILE
       #=====================

overview:
  format-version: 3
  program: TemplotMEC
  version: 2.24.a
  saved: 2019-10-29T03:11:43
  template-count = 26 # comment only      

templates:
  - index = 0
    something = "N 0"
    bignum = 145686781
    sometime = 2019-08-01T02:33:51
    somedesc = "BH • EM  half-diamond K- 8  +  V- 8   RH"
    name = "Downingham EM"
    type = "[slip 023300]  slip half-diamond"
 ...etc ...
   infos:
     - text: memo text ...
   memos:
     - text: memo text ...

  - index = 7
 ...etc ...

The pick of the bunch by far for me. Simple, elegant, very readable and SHORT!

Anyway, just my thoughts. Happy to provide further info but I have to run now.

Cheers,

g






posted: 30 Oct 2019 13:34

from:

Martin Wynne
 
West Of The Severn - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Graeme wrote:
This is not too far removed from the old 'ini' files from DOS days. Basically a simple store of key-value pairs (which is pretty much to be expected for this kind of application).

Field Names
We spoke earlier about readability and its importance. To help this I would strongly suggest having meaningful (and unique) names for the fields as it will make life easier when viewing these files.

For example, lines 1010 and 1015 are both called "s" but one is a date and one is a time.
Hi Graeme,

Thanks for taking the trouble to do all that.

Yes, I think I mentioned that I intended to use the old Windows ini files format, not least because it is fully supported by Delphi and Lazarus in the TStrings.Values function. It is already used for the save program preferences function, and I'm very comfortable with it.

Templot does already include an open-source XML engine, used for the SK9 files for saving the sketchboard (in T2 only at present). But it is much more clunky to use than simple ini files, and the extra functions are not needed for a BOX contents file.

Who's this "we"? :)

I haven't said a word about readability, and I really don't see the need for it -- no-one is going to choose a MECBOX file for bedtime reading. Much more important is human decodability, which is easily provided by reference to the encoding list in the files. That's half the reason for making the code files open-source, surely?

-s , -b , -i , etc. are type indicators not names: s=string, b=boolean, i=integer, etc.

I wrote a utility exe to generate the encoding list and here is the start of it. I shudder to think of the work needed to generate 500 meaningful names for this lot by hand:
     
 mecbox_encoding.pngmecbox_encoding.png

(The delimiters use the ` character, so all strings have that replaced before being saved.)

I don't see any need for the MECBOX format to replace the native OTBOX format for normal use, not least because it is so much slower to save and load than OTBOX (which is in effect a binary memory dump). Templot keeps a rolling backup of every change to the box contents, and it needs to be fast.

MECBOX is simply a means of import/export transfers between T2 and OT/MEC. New users won't have any need for it, and existing users will rarely need to use it more than once for each old BOX file, or a folderful of them.

And having now got it working fine, I don't see much need to change the format?

I will post a scruff release soon, so that users can backup their BOX files in MECBOX format, just in case that bus is running amok today. :)

cheers,

Martin. 

posted: 31 Oct 2019 02:48

from:

Graeme
 
Bangkok - Thailand

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:
Values['1120-b']:=IntToStr(......

Ahhhh - so "1120-b" is actually a field name! From the data, interpreted it as a field called "b" and due to some goofy cut-n-paste foolishness the 1120 line number had come across too. hahaha

Well I did say that coming up with 450 distinct names would be a challenge. :)

Oh well - this track seems full of misunderstandings on my part.
Yes, I think I mentioned that I intended to use the old Windows ini files format, ...
Wellllll .... the message I got was

from 13 Sep 2019 17:22
What is needed is an entirely new file format based on XML or similar. ... If someone wanted to contribute to OpenTemplot, creating a new file export/import format in XML would be a massive step forward.

... then from 17 Oct 2019 20:48
However what does compile in Lazarus is the old-fashioned Windows INI files format, ...  It works just fine, so I'm tempted to use it for a new compatible BOX file format, rather than XML or any of the other fancier suggestions.
This sounded to me that we were going to an as-yet-undecided text format for THE BOX file format - i.e. both T2 and OT would use it so that each could read the files of the other.

Had I realised that the INI file was a fait accompli I would not have bothered with all of this.

BUT ... it is all moot because I had missed perhaps the most vital point ...
I don't see any need for the MECBOX format to replace the native OTBOX format for normal use,
SO ... the plan is to keep both BOX and OTBOX as the native formats for T2 and OT and use the text-based (INI) format purely for interchange.

OK. C'est la vie! Onward and upward....

g

posted: 31 Oct 2019 06:04

from:

Martin Wynne
 
West Of The Severn - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Graeme wrote:
Had I realised that the INI file was a fait accompli I would not have bothered with all of this.
Hi Graeme,

I'm very sorry if you feel I have misled you in some way.

Rob's post that he was using TemplotMEC in anger made me realise that an interchange solution was needed with some urgency. When I discovered that I was making more progress with a text-based export/import format than I expected, I did suggest to you on 20th October:
Please don't spend any time on a new file format, or at least not yet. I've spent the last few days on that and I'm close to having something usable. If you are looking for something to have a go at, how about the file viewer function?

I think perhaps you missed the the wording of my previous post:
What is needed is an entirely new file format based on XML or similar. ... If someone wanted to contribute to OpenTemplot, creating a new file export/import format in XML would be a massive step forward.
The essential words there were "export/import", rather than a new native save/load file format.

There is still a great need for that, because the existing BOX (and now OTBOX) formats are basically just binary dumps in a format I cobbled together 25 years ago without any thought or knowledge of future developments or needs. Fortunately I left some significant areas of empty space in the type definitions, which over the years I have been filling up with an assortment of new settings as the program developed. Which means that the format has now become a shambolic mess which is in great need of replacement. But it works and it's fast, and there are by now thousands of files around the world in that format, so there has always been a good case for IIABDFI.

The shortcomings of such a format became obvious when I tried to recreate it in Lazarus for OT/MEC. In an effort to be cross-platform, Lazarus maps the Intel 80-bit Extended floating point format as the IEEE 64-bit Double format:

 http://en.wikipedia.org/wiki/Double-precision_floating-point_format

 http://en.wikipedia.org/wiki/Extended_precision

Consequently all the type definitions in the Lazarus files are shorter and incompatible with the original BOX format from Delphi. I was forced to differentiate it as a new OTBOX format.

If a way could be found around that, while retaining a fast binary format, there would be no need for a much slower text-based interchange format. I spent a lot of time trying to find such a solution, with only limited success. It should be possible because the conversions are built-in to the CPU registers:

"The IA32, x86-64, and Itanium processors support an 80-bit "double extended" extended precision format with a 64-bit significand. The Intel 8087 math coprocessor was the first x86 device which supported floating point arithmetic in hardware. It was designed to support a 32-bit "single precision" format and a 64-bit "double precision" format for encoding and interchanging floating point numbers. The temporary real (extended) format was designed not to store data at higher precision as such, but rather primarily to allow for the computation of double results more reliably and accurately ... ... The 8087 automatically converts numbers to this format when loading floating point registers from memory and also converts results back to the more conventional formats when storing the registers back into memory. ... ... The floating-point unit (FPU) on all subsequent x86 processors have supported this format. As a result software can be developed which takes advantage of the higher precision provided by this format."

1143px-X86_Extended_Floating_Point_Format.svg.png1143px-X86_Extended_Floating_Point_Format.svg.png
wikimedia commons

But we are where we are. One of the reasons I was doubtful about going the open-source route for Templot is that I have a feeling I'm on a different page in the hymn book from everyone else. :?

cheers,

Martin.

posted: 31 Oct 2019 09:23

from:

Graeme
 
Bangkok - Thailand

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:
I'm very sorry if you feel I have misled you in some way.

Hi Martin,

No, I am afraid I let my enthusiasm get the better of me and saw what I wanted to see.  :D

With my new understanding of file formats I am once again fully focused on the file viewer (where I should have been all along. :roll: )

But we are where we are. One of the reasons I was doubtful about going the open-source route for Templot is that I have a feeling I'm on a different page in the hymn book from everyone else. :?
Generally a GOOD thing, if you ask me. :)


Cheers,

g





Last edited on 31 Oct 2019 09:24 by Graeme
posted: 31 Oct 2019 21:38

from:

Rob Manchester
 
Manchester - United Kingdom

click the date to link to this post
click member name to view archived images
view images in gallery view images as slides
Martin Wynne wrote:


But we are where we are. One of the reasons I was doubtful about going the open-source route for Templot is that I have a feeling I'm on a different page in the hymn book from everyone else. :?

cheers,

Martin.
Hi Martin,
If you share a hymm book like we used to do in church it makes sure you are singing the same song as 1 other person - mind you the rest of the congregation maybe be singing something entirely different :D
Rob




Templot Club > Forums > TemplotMEC nuts and bolts > File formats for T3/MEC
about Templot Club

Templot Companion - User Guide - A-Z Index Templot Explained for beginners Please click: important information for new members and first-time visitors.
indexing link for search engines

back to top of page


Please read this important note about copyright: Unless stated otherwise, all the files submitted to this web site are copyright and the property of the respective contributor. You are welcome to use them for your own personal non-commercial purposes, and in your messages on this web site. If you want to publish any of this material elsewhere or use it commercially, you must first obtain the owner's permission to do so.
The small print: All material submitted to this web site is the responsibility of the respective contributor. By submitting material to this web site you acknowledge that you accept full responsibility for the material submitted. The owner of this web site is not responsible for any content displayed here other than his own contributions. The owner of this web site may edit, modify or remove any content at any time without giving notice or reason. Problems with this web site? Contact webmaster@templot.com.   This web site uses cookies: click for information.  
© 2020  

Powered by UltraBB - © 2009 Data 1 Systems