View source for Dev:FileConventions


== Conventions for {{rml|R=no}} files ==

=== Which file name extension shall a {{rml|R=no}} file get? ===

<nowiki>*.xml</nowiki> or <nowiki>*.railml</nowiki>, that is the question. The answer is: Both. It was not a convention from the beginning, but best practice should be:
* '''<nowiki>*.railml</nowiki>''' shall only be used for files / exports from tools which are certified by {{org}} commitee. Being a registered trademark, ''{{rml}}'' should be a kind of quality sign. As a certification is mandatory since the release of {{rml|2.2}}, all tools in productive use shall use '''<nowiki>*.railml</nowiki>''' for files / exports.
* '''<nowiki>*.xml</nowiki>''' is always allowed since {{rml}} files are xml files. It shall be used for non-certified or experimental files. Since the certification process started in 2013 all {{rml}} version 1.x files / exports must use this extension. {{rml}} version 2.0 or 2.1 files / exports shall use '''<nowiki>*.xml</nowiki>''' extension, if the programme was delivered before July 1st 2013 and is not certified up to now.
* '''<nowiki>*.railmlx</nowiki>''': compressed <nowiki>*.railML</nowiki> files shall use this extension if compressed implicitly by a certified software. Manually compressed files, of course, may have the extension <nowiki>*.zip</nowiki>.

===How to interpret missing attributes and elements in the file?===

If an attribute or an element is missing in a railML file, it shall be read as "not known".

== Technical description ==

=== Compressed {{rml|R=no}} files ===
Due to the large size of {{rml}} files by trend - as being text files with all their inefficiencies - file compression has been introduced with {{rml}} 2.2. The standard of compression of {{rml}} files settles to the following:

# {{rml}} files shall be compressed using Deflate compression algorithm and packed into a ZIP file archive corresponding to {{external|https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT|.ZIP File Format Specification|category=EN}}
# The advantage of Deflate and ZIP to allow compression and decompression ‘on the fly’ shall fully be supported.
# The {{rml}} (xml) file shall be the first file of the ZIP archive. 
# The original file name of the ZIP archive (railmlx file) shall be identical to the file name of the {{rml}} file in the ZIP archive - only the file extensions shall be different. The file extension of the packed {{rml}} file shall be *.xml or *.railml as stated above, the file extension of the ZIP archive shall always be *.railmlx.
# If the {{rml}} file uses non-{{rml}} namespaces with their schema files (XSD files) not available at an Internet URL, these schema files shall be packed into the ZIP file 'behind' the {{rml}} file. All files in the archive should validate without any further files other than:
::* {{rml}} XSD schema files
::* Dublin Core XSD schema files
::* (MathML XSD schema files)
: If this way is used, the attribute {{Attr|xsi:schemaLocation}} shall contain the XSD file name as packed into the ZIP archive without any path nor prefix, so without <nowiki>'http(s)://'</nowiki> nor <nowiki>'urn:'</nowiki> nor such and without any delimiter /. See also [[dev:catalog#Defining_namespaces_and_validating_{{rml|R=no|link=no}}_files | Defining namespaces and validating {{rml}} files]] and [[dev:usingAny#Namespace_handling | Namespace handling]].

Please note the following further remarks:
* For the {{rml}} file to be the ''first'' in the ZIP archive (rule 3), it is necessary to be the first Central File Header in the Central Directory which is at the ''end'' of the ZIP archive. It does not need to be the first file entry in the data stream itself - that is, its value ''Relative offset of local file header'' does not need to be zero. (In a ZIP file, the ''Central Directory'' at the end of the file is read first.)
* The railmlx file must start with a Local File Header Signature (4 Bytes h50 h4B h03 h04) - no arbitrary data before the first Local File Header is allowed. (An End of Central Directory Signature h50 h4B h05 h06 would also be possible in case of an empty ZIP file but does not make sense here.) This is to allow detection whether a {{rml}} file is compressed or not - no valid uncompressed {{rml}} file can start with these bytes (see also below).
* railmlx files must not use a ZIP File Comment in the End of Central Directory Record (EOCD) nor any arbitrary data behind the EOCD. To correctly read a ZIP file from its end, the reading software has to find the start of the EOCD first. With ZIP File Comment, this could only be done by scanning for the EOCD Signature backwards from the end. But strangely, it is not secured that no EOCD Signature is part of the ZIP File Comment. This seems to be a weakness of the ZIP file format. To provide a comment in a railmlx file, use File Comment of a Central Directory File Header instead of a ZIP File Comment.
* railmlx files must not be spread over several ‘disks’ - the values ''Number of this disk'' and ''Disk where central directory starts'' of EOCD always have to be zero. 
* To fully allow compressing and uncompressing ‘on the fly’, the values CRC32 and CompressedSize of the Local File Headers in the ZIP archive does not need to be used. A reading (uncompressing) software shall ignore these values (since they may be unknown to the writing software when writing the Local File Headers on the fly). However, the corresponding values CRC32 and CompressedSize of the Central Directory File Headers must be correctly set.

=== Encoding of {{rml|R=no}} files ===

{{rml}} files shall be encoded UTF-8. This means, the attribute {{Attr|encoding}} of the XML declaration shall be set to <code>encoding='UTF-8'</code>:

<code>
<?xml version="1.0" encoding="UTF-8" ?>
</code>

Please note that this only defines the encoding ''of the XML ({{rml}}) file''. This does not necessarily be identical with the original encoding ''of the data (contents) of the file''. The term ''data'' here (concerning {{rml}} files) often applies to proper names of stations and such.

If the data is not also (by coincidence?) from UTF-8, it has to be re-coded into UTF-8 to be stored in the {{rml}} file. In most cases nowadays, data will come from UTF-16 or any other Unicode format which can be re-coded into UTF-8 and vice-versa without loss.

Also, any 8 bit code page (Single Byte Character Set, SBCS) can be re-coded into UTF-8 but here comes the only problem which may happen: If data from a {{rml}} file has to be stored in a SBCS, this may not be possible without loss as long as the original code page is not known. Therefore, it is recommended to note the origin of the data whenever possible using the field <tt><[[Dev:catalog#Dublin Core|dc:language]]></tt> whenever possible.

In most cases of proper names of stations, a software will not know the ''language'' of station names (since station names normally are international and do not belong to a certain language). To tell the character set anyway - without a language - use the element <tt><[[Dev:catalog#Dublin Core|dc:language]]></tt> as follows, with the language sub-tag set to <code>und</code> for ''undefined'' but the script sub-tag set to the original character set:<br>
:<code><dc:language>und-Latn</dc:language>	for Latin character set</code><br>
:<code><dc:language>und-Grek</dc:language>	for Greek character set</code><br>
:<code><dc:language>und-Cyrl</dc:language>	for Cyrillic character set</code><br>
:<code><dc:language>und-Arab</dc:language>	for Arabic character set</code><br>
:<code><dc:language>und-Hebr</dc:language>	for Hebrew character set</code><br>
:<code><dc:language>und-Japn</dc:language>	for Japanese character set</code><br>
and so on.

=== XML declaration ===

Since <code>encoding='UTF-8'</code> is the default value for {{Attr|encoding}}, and the XML declaration itself is optional, too, one could mean that it is ok to skip the entire XML declaration. This would mean a valid {{rml}} starting with 
<code><railml version…</code> rather than <code><?xml version='1.0' encoding='UTF-8' ?></code>.

It is strongly recommended and best practice, '''not''' to omit the XML declaration. Always remember that file names and their extensions may be altered, so it should easily be possible to validate a file as being of XML format. Also, no arbitrary data are allowed before the XML declaration (except BOMs, see below).

(Please note that this article applies on {{rml}} ''files''. It may be ok to omit the XML declaration if {{rml}} is encapsulated by, or embedded in, a higher protocol level and thus is part of a data stream which is not a file. Apart from such special use cases, {{rml}} is a file format for general data exchange. In this general meaning - with no precise data source and destination - the XML declaration is not to be omitted.)

However, you can omit the attribute {{Attr|encoding}} if it is UTF-8 (but not if the file is from any other code page). The declaration of encoding is also case-sensitive (both upper-case and lower-case letters are allowed), but it is recommended to use upper-case "UTF-8" if it is not omitted.

=== Byte Order Mark ===

A {{rml}} file may start with a {{wiki|Byte_order_mark|Byte Order Mark}} (BOM) but since no special byte order applies to UTF-8, it is not necessary and therefore not recommended.

So, a valid {{rml}} file may start with the following byte sequences:
* '''<?xml          '''	- no BOM, valid XML declaration: {{Attr|encoding}} shall be interpreted or file must be encoded UTF-8 if there is no encoding given
* '''hEF hBB hBF    '''	- BOM for UTF-8: {{Attr|encoding}} must be set to UTF-8 or omitted
* '''h50 h4B h03 h04'''	- compressed {{rml}} file: decompress and unpack the file and go on reading BOM and/or {{Attr|encoding}} of the decompressed data

And possibly:
* '''<railml        '''	- no BOM, no XML declaration: file must be encoded UTF-8. Since this is against best practice, you may refuse such files.

The following byte sequences may be valid XML files but unsupported by {{rml}}:
* '''h00 h00 hFE hFF'''	- UCS-4, big-endian (1234 order)
* '''hFF hFE        '''	- UCS-4, little-endian (4321 order) or UTF-16, little-endian
* '''h00 h00 hFF hFE'''	- UCS-4, unusual octet order (2143)
* '''hFE hFF        '''	- UCS-4, unusual octet order (3412) or UTF-16, big-endian

[[category:Common Concept]]
{{interwiki}}