Creating a Database Subset
Introduction
NOTE: This page explains the manual configuration and method of performing subsetting. It is recommended to utilize the ClipmapSubsetCopier tool referenced from: ClipmapSubsetCopier
Aechelon provides the ability to subset our databases. This may be useful in cases where storage or copy times are a limitation.
ClipCopy
ClipCopy is the command line tool used to subset database. The tool takes in a database layer clipmap, a kml file describing the region and resolution of the area, and outputs a smaller version of the database layer clipmap.
ClipCopy Usage
The ClipCopy tools comes installed with every software release and can be found under
E:\CM\software\<version>\bin\optimized-x64
To run the tool, open a command window and navigate to to the data folder
ClipCopy Arguments
The ClipCopy tool has several arguments available. Below is a list of all arguments needed to create a subset. For a full listing of all available arguments please type clipcopy.exe --help.
--in
The .ct file of the database layer
--out
The output folder of the resulting subset database layer. This will produce a clipmap and a .ct file.
--vectorSubset
The kml file describing the bounding box and resolution of the subset
--hashClipmap
Creates a resulting clipmap with built in hash data. Always include this argument
--ctRoot
The path to the root of the input clipmaps. Since most clipmap .ct files contain the CT_ROOT variable this allows the tool to resolve the proper path to the source clipmap.
--estimateSize
Estimates the size of the resulting clipmap without creating the output.
ClipCopy Command
In the following example ClipCopy is used to make a subset of the out-the-window imagery of the database.
E:\CM\software\<version>\clipCopy.exe --in <DATABASE_OVERLAY_PATH>\DB_TERRAIN\otw.ct --out <SUBSET_OUTPUT_FOLDER>\GLOBAL_OTW\OTW --vectorSubset <KML_FILE> --hashClipmap --ctRoot E:\Clipmaps
The resulting output is a folder with the imagery along with a .ct file.
The only database layer than cannot be subset is the geo specific textures, called Paged Textures. This layer should be copied whole to the subset.
ClipAux Command
For some data layers it is necessary to run one final command on the subset output to create the auxiliary metadata. This is done using the command line tool, clipAuxFile which is also delivered with all runtime software.
clipAuxFile.exe --tilemapwriteAll --clipmap <ct file path>.ct
The following data layers require this command to be run on the subset output clipmaps.
featureShadows.ct
lightMap.ct
littoral.ct
materials.ct
otw.ct
terrain_mesh.ct
No additional output is created from this command as the data is written directly into the clipmaps.
Running the Subset
Once all database layers are subset, the user can run the subset using the original source database overlay configuration files. The only change will be to point to the root of the new clipmap location. This can be done using the following tokens with paths to the subset clipmap location:
PagedFeatureCtRoot E:\Clipmaps\Subset
TerrainTextureCtRoot E:\Clipmaps\Subset
TerrainMeshCtRoot E:\Clipmaps\Subset
ClipCopy Script Example
Below is an example script to run clipcopy on all available layers in the database as well as copy the paged textures to the subset. There are several variables that can be set at the beginning of the script to define path locations.
@ECHO OFF
E:
cd E:\CM\data
REM --------------
REM --- INPUTS ---
REM --------------
rem --- KML_FILE: The kml file that defines the subset boundary area ---
set KML_FILE=test.kml
rem --- OUTPUT_FOLDER: The location of the subset clipmap output ---
set OUTPUT_FOLDER=E:\CLIPMAPS\SUBSET
rem --- CT_ROOT: The prefix path of the source clipmaps. The .ct files use this token. ---
set CT_ROOT=E:\CLIPMAPS
rem --- DATABASE_OVERLAY_PATH: The overlay path where all of the .ct files can be found which point to the source clipmaps ---
set DATABASE_OVERLAY_PATH=E:\CM\data\DB_OVERLAY\DB_GLOBAL_COMM_V7\RELEASE\V7_1\DB
REM ---------------
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_TERRAIN\featureShadows.ct --out %OUTPUT_FOLDER%\GLOBAL_COMM_TF_SHADOWS\FEATURE_SHADOWS --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_TERRAIN\lightMap.ct --out %OUTPUT_FOLDER%\GLOBAL_COMM_OTW\LIGHT_MAP --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_TERRAIN\littoral.ct --out %OUTPUT_FOLDER%\GLOBAL_COMM_OTW\LITTORAL --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_TERRAIN\materials.ct --out %OUTPUT_FOLDER%\GLOBAL_COMM_MAT\MATERIALS --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_TERRAIN\otw.ct --out %OUTPUT_FOLDER%\GLOBAL_COMM_OTW\OTW --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_TERRAIN\terrain_mesh.ct --out %OUTPUT_FOLDER%\GLOBAL_COMM_TER\TERRAIN_MESH --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\paged_animtree_data.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\animtree_data --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\paged_aero_light_data.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\aero_light_data --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\paged_feat_data.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\feat_data --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\paged_grass_data.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\grass_data --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\paged_instanced_data.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\instanced_data --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\paged_light_data.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\light_data --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\paged_railroad_data.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\railroad_data --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\paged_road_data.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\road_data --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\paged_tree_data.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\tree_data --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\paged_wire_data.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\wire_data --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\pagedHie.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\bakedbins_data_singlepass --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\multiPassPagedHie.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\bakedbins_data_multipass --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\radar_multiPassPagedHie.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\radar_bakedbins_data_multipass --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipCopy.exe --in %DATABASE_OVERLAY_PATH%\DB_PAGED_FEATURES\radar_pagedHie.ct --out %OUTPUT_FOLDER%\DB_PAGED_FEATURES\radar_bakedbins_data_singlepass --vectorSubset %KML_FILE% --hashClipmap --ctRoot %CT_ROOT%
rem Need to copy textures clipmap directly since it doesn't get subset
xcopy /e /v /i %CT_ROOT%\V7_1\DB_PAGED_FEATURES\bakedbins_data_textures %OUTPUT_FOLDER%\DB_PAGED_FEATURES\bakedbins_data_textures
xcopy /e /v /i %CT_ROOT%\V7_1\DB_PAGED_FEATURES\radar_bakedbins_data_textures %OUTPUT_FOLDER%\DB_PAGED_FEATURES\radar_bakedbins_data_textures
rem Write Aux File Data to Clipmaps
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipAuxFile.exe --tilemapwriteAll --clipmap %OUTPUT_FOLDER%\GLOBAL_COMM_TF_SHADOWS\FEATURE_SHADOWS\featureShadows.ct
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipAuxFile.exe --tilemapwriteAll --clipmap %OUTPUT_FOLDER%\GLOBAL_COMM_OTW\LIGHT_MAP\lightMap.ct
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipAuxFile.exe --tilemapwriteAll --clipmap %OUTPUT_FOLDER%\GLOBAL_COMM_OTW\LITTORAL\littoral.ct
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipAuxFile.exe --tilemapwriteAll --clipmap %OUTPUT_FOLDER%\GLOBAL_COMM_MAT\MATERIALS\materials.ct
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipAuxFile.exe --tilemapwriteAll --clipmap %OUTPUT_FOLDER%\GLOBAL_COMM_OTW\OTW\otw.ct
E:\CM\software\%AECHELON_SOFTWARE_DIR%\clipAuxFile.exe --tilemapwriteAll --clipmap %OUTPUT_FOLDER%\GLOBAL_COMM_TER\TERRAIN_MESH\terrain_mesh.ct
KML File
A kml file is needed to provide the subset definition of the area. The most basic definition is a square bounding box. If this is provided to the tool it will output data for content inside of the bounding box and discard everything else.
Defining Resolution in the KML File
By default if a bounding area is defined, then the resolution output by ClipCopy will be the highest available for the database in that area. If the user wishes to specify a specific, usually lower, resolution then they would add resolution=<value> to the description.
<description>resolution=64m</description>
KML File Example
Below is an example KML file which defines a high resolution area around Japan and South Korea. It also defines low resolution areas (64m) around the entire globe to provide some content in those areas.
<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2" xmlns:gx="http://www.google.com/kml/ext/2.2" xmlns:kml="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom ">
<Document>
<name>Subset Definition Example</name>
<open>1</open>
<StyleMap id="msn_ylw-pushpin00">
<Pair>
<key>normal</key>
<styleUrl>#sn_ylw-pushpin0</styleUrl>
</Pair>
<Pair>
<key>highlight</key>
<styleUrl>#sh_ylw-pushpin0</styleUrl>
</Pair>
</StyleMap>
<StyleMap id="msn_ylw-pushpin000">
<Pair>
<key>normal</key>
<styleUrl>#sn_ylw-pushpin</styleUrl>
</Pair>
<Pair>
<key>highlight</key>
<styleUrl>#sh_ylw-pushpin</styleUrl>
</Pair>
</StyleMap>
<Style id="sh_ylw-pushpin">
<IconStyle>
<scale>1.3</scale>
<Icon>
<href><http://maps.google.com/mapfiles/kml/pushpin/ylw-pushpin.png</href>>
</Icon>
<hotSpot x="20" y="2" xunits="pixels" yunits="pixels"/>
</IconStyle>
<LineStyle>
<color>ffff5500</color>
<width>8</width>
</LineStyle>
<PolyStyle>
<fill>0</fill>
</PolyStyle>
</Style>
<Style id="sh_ylw-pushpin0">
<IconStyle>
<scale>1.3</scale>
<Icon>
<href><http://maps.google.com/mapfiles/kml/pushpin/ylw-pushpin.png</href>>
</Icon>
<hotSpot x="20" y="2" xunits="pixels" yunits="pixels"/>
</IconStyle>
<LineStyle>
<color>ffff5500</color>
<width>8</width>
</LineStyle>
<PolyStyle>
<fill>0</fill>
</PolyStyle>
</Style>
<Style id="sn_ylw-pushpin">
<IconStyle>
<scale>1.1</scale>
<Icon>
<href><http://maps.google.com/mapfiles/kml/pushpin/ylw-pushpin.png</href>>
</Icon>
<hotSpot x="20" y="2" xunits="pixels" yunits="pixels"/>
</IconStyle>
<LineStyle>
<color>ffff5500</color>
<width>8</width>
</LineStyle>
<PolyStyle>
<fill>0</fill>
</PolyStyle>
</Style>
<Style id="sn_ylw-pushpin0">
<IconStyle>
<scale>1.1</scale>
<Icon>
<href><http://maps.google.com/mapfiles/kml/pushpin/ylw-pushpin.png</href>>
</Icon>
<hotSpot x="20" y="2" xunits="pixels" yunits="pixels"/>
</IconStyle>
<LineStyle>
<color>ffff5500</color>
<width>8</width>
</LineStyle>
<PolyStyle>
<fill>0</fill>
</PolyStyle>
</Style>
<StyleMap id="m_ylw-pushpin">
<Pair>
<key>normal</key>
<styleUrl>#s_ylw-pushpin</styleUrl>
</Pair>
<Pair>
<key>highlight</key>
<styleUrl>#s_ylw-pushpin_hl</styleUrl>
</Pair>
</StyleMap>
<Style id="s_ylw-pushpin">
<IconStyle>
<scale>1.1</scale>
<Icon>
<href><http://maps.google.com/mapfiles/kml/pushpin/ylw-pushpin.png</href>>
</Icon>
<hotSpot x="20" y="2" xunits="pixels" yunits="pixels"/>
</IconStyle>
<LineStyle>
<color>ff0000ff</color>
<width>4</width>
</LineStyle>
<PolyStyle>
<fill>0</fill>
</PolyStyle>
</Style>
<Style id="s_ylw-pushpin_hl">
<IconStyle>
<scale>1.3</scale>
<Icon>
<href><http://maps.google.com/mapfiles/kml/pushpin/ylw-pushpin.png</href>>
</Icon>
<hotSpot x="20" y="2" xunits="pixels" yunits="pixels"/>
</IconStyle>
<LineStyle>
<color>ff0000ff</color>
<width>4</width>
</LineStyle>
<PolyStyle>
<fill>0</fill>
</PolyStyle>
</Style>
<Placemark>
<name>Japan_South_Korea_Subset</name>
<styleUrl>#msn_ylw-pushpin000</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
122.6958677190726,22.94364356561665,0 145.0028407236265,23.37701953795413,0 149.0130919501049,45.84496795510012,0 117.3470825263173,44.82655235807989,0 122.6958677190726,22.94364356561665,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Folder>
<name>Low_Res_Background_64m</name>
<Placemark>
<name>low_res1</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
0,-85,0 60,-85,0 60,0,0 0,0,0 0,-85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res2</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
0,85,0 60,85,0 60,0,0 0,0,0 0,85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res3</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
0,85,0 -60,85,0 -60,0,0 0,0,0 0,85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res4</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
-60,85,0 -120,85,0 -120,0,0 -60,0,0 -60,85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res5</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
-120,85,0 -180,85,0 -180,0,0 -120,0,0 -120,85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res6</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
0,-85,0 -60,-85,0 -60,0,0 0,0,0 0,-85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res7</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
-60,-85,0 -120,-85,0 -120,0,0 -60,0,0 -60,-85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res8</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
-120,-85,0 -180,-85,0 -180,0,0 -120,0,0 -120,-85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res9</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
60,85,0 120,85,0 120,0,0 60,0,0 60,85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res10</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
120,85,0 180,85,0 180,0,0 120,0,0 120,85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res11</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
60,-85,0 120,-85,0 120,0,0 60,0,0 60,-85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
<Placemark>
<name>low_res12</name>
<description>resolution=64m</description>
<styleUrl>#m_ylw-pushpin</styleUrl>
<Polygon>
<tessellate>1</tessellate>
<outerBoundaryIs>
<LinearRing>
<coordinates>
120,-85,0 180,-85,0 180,0,0 120,0,0 120,-85,0
</coordinates>
</LinearRing>
</outerBoundaryIs>
</Polygon>
</Placemark>
</Folder>
</Document>
</kml>