opal.parser package

Notebooks

• OptimizerParser
  • Load names of design variables and objectives
  • Find number of generations run by optimizer
  • Load all data in first generation
  • Assign labels, data, and plot paralllel coord
  • Plot parallel coord plot and open in new tab:

opal.parser.FieldParser module

class opal.parser.FieldParser.FieldParser

Bases: BaseParser

Read OPAL field data generated with compile flag DBG_SCALARFIELD enabled.

_columns

the keys are the column names and the values their units

Type:

dict

_df

the data

Type:

pandas.core.frame.DataFrame

_dim

list of number of grid points per dimension

Type:

list

_stride

list of stride per dimension

Type:

list

__init__()

Constructor

_check_header(filename)

_parse_header(filename)

clear()

Reset attributes

property dataframe

Get all data.

Returns:

all the data

Return type:

pandas.core.frame.DataFrame

property dimension

returns: the number of grid points per dimension :rtype: list

property field

returns: the field data on the grid :rtype: numpy.ndarray

get_unit_dictionary()

Get the dictionary of units.

Returns:

column names (keys) and their units (values)

Return type:

dict

property image

returns: the image potential (if available) :rtype: numpy.array

property indices

returns: the grid point indices :rtype: numpy.ndarray

is_scalar()

Returns:

True if a scalar field

Return type:

bool

parse(filename)

Parse a file.

Parameters:

filename (str) – the full name of a file to parse

property positions

returns: the positions at the grid points :rtype: numpy.ndarray

property stride

returns: the stride per dimension :rtype: list

opal.parser.H5Error module

exception opal.parser.H5Error.H5AttributeError(msg)

Bases: H5Error

Exception raised for errors concerning attributes.

__init__(msg)

exception opal.parser.H5Error.H5DatasetError(msg)

Bases: H5Error

Exception raised for errors concerning datasets.

__init__(msg)

exception opal.parser.H5Error.H5Error(msg)

Bases: Exception

Base class for H5 exceptions. (see also https://docs.python.org/2/tutorial/errors.html)

__msg

explanation of the error

Type:

str

__init__(msg)

exception opal.parser.H5Error.H5OverflowError(msg)

Bases: H5Error

Exception raised for errors in accessing data.

__init__(msg)

exception opal.parser.H5Error.H5ParseError(msg)

Bases: H5Error

Exception raised for errors in parsing.

__init__(msg)

opal.parser.H5Parser module

class opal.parser.H5Parser.H5Parser

Bases: BaseParser

__init__()

Constructor.

_check_step(step)

clear()

Clear data.

close()

Closes h5 file.

getGlobalAttribute(attrName)

Obtain value of global attribute.

Parameters:

attrName (str) – Name of global attribute

Returns:

Value as string.

Return type:

str

getGlobalAttributes()

All attributes of root.

Returns:

A list of strings of all root attributes

Return type:

list

getNSteps()

Number of steps in file.

Returns:

The number of steps

Return type:

int

getStepAttribute(attrName, step=0)

Obtain the attribute data of a step.

Parameters:

• attrName (str) – Name of attribute

• step (int, optional) – Steps of file

getStepAttributes(step=0)

Attributes names of a step.

Parameters:

step (int, optional) – Steps of file

Returns:

All attribute names of a step

Return type:

list [str]

getStepDataset(dsetName, step=0)

Obtain the data of a step.

Parameters:

• dsetName (str) – Name of dataset

• step (int, optional) – Steps of file

Returns:

Array of a dataset of a step (check with getStepDatasets())

Return type:

numpy.ndarray, (N,)

getStepDatasets(step=0)

All step dataset names.

Parameters:

step (int, optional) – Steps of file

Returns:

All attribute names of a step

Return type:

list [str]

parse(fname, info=False)

Read a H5 file generated by OPAL

Parameters:

• fname (str) – H5 filename

• info (bool, optional) – Print H5 file summary

opal.parser.HistogramParser module

class opal.parser.HistogramParser.HistogramParser

Bases: BaseParser

Parses OPAL *.hist files

Parses *.hist files of OPAL that are written for probe elements and *.hist files of measurements.

The first line in both files is a comment line starting with ‘#’. The measurement header line should be:

# Histogram bin counts (radius ['unit'], intensity ['unit'])

where a simulation header is:

# Histogram bin counts (min, max, nbins, binsize) 'min' 'unit' 'max' 'unit' '#bins' 'binsize' 'unit'

_info

Dictionary with histogram info:

‘min’

List of minimum value and unit

‘max’

List of maximum value and unit

‘nbins’

List of number of bins and unit

‘binsize’

List of size of bin and unit

‘bincount’

List of bin counts and unit

Type:

dict

__init__()

clear()

Clear data.

getDataOfVariable(var)

getUnitOfVariable(var)

getVariables()

isVariable(var)

parse(filename)

property size

opal.parser.LatticeParser module

class opal.parser.LatticeParser.LatticeParser

Bases: object

__init__()

Constructor.

plot(lfile, fig, ax1, ax2)

opal.parser.LossParser module

class opal.parser.LossParser.LossParser

Bases: BaseParser

__init__()

clear()

Clear data.

getDataOfVariable(var)

getUnitOfVariable(var)

getVariableNames()

isVariable(var)

parse(filename)

Parser method for loss files

Examples

1. Clean

(OPAL version <= 2.4) line = “# Element STQ1 x (m), y (m), z (m), px ( ), py ( ), pz ( ), id, turn, time (ns)” (OPAL version >= 2.6) line = “# x (m), y (m), z (m), px ( ), py ( ), pz ( ), id, turn, time (s)”

gets

line = ” x (m), y (m), z (m), px ( ), py ( ), pz ( ), id, turn, time (s)”

2. no commas

line = ” x (m), y (m), z (m), px ( ), py ( ), pz ( ), id, turn, time (s)”

gets

line = ” x (m) y (m) z (m) px ( ) py ( ) pz ( ) id turn time (s)”

property size

opal.parser.OptimizerParser module

class opal.parser.OptimizerParser.OptimizerParser

Bases: BaseParser

Parsing of JSON files generated by the OPAL optimizer

__ids

List of all individual IDs of a generation

Type:

list

__dvar_values

List of all design variables values for each ID (order corresponds to __ids)

Type:

numpy.matrix

__obj_values

List of all objective values for each ID (order corresponds to __ids)

Type:

numpy.matrix

__obj_names

Specifies columns in self.__obj_values

Type:

list

__dvar_names

Specifies columns in self.__dvar_values

Type:

list

__pareto

Type:

str

Notes

Supports following JSON files:

{
    "name": "opt-pilot",
    "dvar-bounds": {
        "benergy": [ 0.071, 0.072 ],
        "phiinit": [ 106, 114 ],
        "prinit": [ -0.02, -0.01 ],
        "rinit": [ 2000, 2060 ]
    }
    ,
    "constraints": [
        "dpeak1 > 0.0",
        "dpeak2 > 0.0"
    ]
    ,
    "solutions": [
    {
        "ID": 0,
        "obj":
        {
                "dpeak1": 10.37,
                "dpeak2": 5.43,
                "dpeak3_5": 14.1095
        }
        ,
        "dvar":
        {
                "benergy": 0.0718672,
                "phiinit": 113.653,
                "prinit": -0.0191565,
                "rinit": 2018.26
        }
    }
    ,
    {
        "ID": 18,
        "obj":
        {
                "dpeak1": 5.95,
                "dpeak2": 4.75,
                "dpeak3_5": 17.6668
        }
        ,
        "dvar":
        {
                "benergy": 0.0715704,
                "phiinit": 110.517,
                "prinit": -0.0122177,
                "rinit": 2035.32
        }
    }
    ,
    {
        "ID": 2,
        "obj":
        {
                "dpeak1": 1.84,
                "dpeak2": 6.04,
                "dpeak3_5": 19.2461
        }
        ,
        "dvar":
        {
                "benergy": 0.0711942,
                "phiinit": 107.097,
                "prinit": -0.0198867,
                "rinit": 2034.75
        }
    }
    ]
}

{
    "name": "opt-pilot",
     "OPAL version": "2.0.0",
     "git revision": "1849b7e5130657e8be50d524de0f6c50134f330a",
     "dvar-bounds": {
         "MX": "[ 16, 32 ]",
         "nstep": "[ 10, 40 ]"
     },
     "constraints": [
         "statVariableAt('rms_x',0.0) > 0",
         "statVariableAt('rms_x',0.0) > 0.001"
     ],
     "population": {
         "0": {
             "obj": {
                 "dpeak1": 10.37,
                 "dpeak2": 5.43,
                 "dpeak3_5": 14.1095
             },
             "dvar": {
                 "benergy": 0.0718672,
                 "phiinit": 113.653,
                 "prinit": -0.0191565,
                 "rinit": 2018.26
             }
         },
         "18": {
             "obj": {
                 "dpeak1": 5.95,
                 "dpeak2": 4.75,
                 "dpeak3_5": 17.6668
             },
             "dvar": {
                 "benergy": 0.0715704,
                 "phiinit": 110.517,
                 "prinit": -0.0122177,
                 "rinit": 2035.32
             }
         },
         "2": {
             "obj": {
                 "dpeak1": 1.84,
                 "dpeak2": 6.04,
                 "dpeak3_5": 19.2461
             },
             "dvar": {
                 "benergy": 0.0711942,
                 "phiinit": 107.097,
                 "prinit": -0.0198867,
                 "rinit": 2034.75
             }
         }
     }
 }

Examples

>>> import OptimizerParser as optreader
>>> import sys
>>> try:

1. Find all .json files of a directory, e.g. “./”

>>> optjson = optreader.OptimizerParser()
>>> optjson.parse("./")

2. Read in a generation file, e.g. 1

>>> optjson.readGeneration(1)

3. Get all design variables

>>> dvars = optjson.getDesignVariables()
>>> print ( "Design variables: ", dvars )

4. Get design variable bounds

>>> bounds = optjson.getBounds()
>>> print ( "Bounds: ", bounds )

4a. Get only bounds of a specific design variable

>>> bound = optjson.getBounds(dvars[2])
>>> lower = bound[0]
>>> upper = bound[1]
>>> print ( "dvar: " + dvars[1] + "
>>>     " lower bound: " + str(lower) + "
>>>     " upper bound: " + str(upper))

5. Get all objectives

>>> objs = optjson.getObjectives()
>>> print ( "Objectives: ", objs )

6. Get all constraints

>>> constr = optjson.getConstraints()
>>> print ("Constraints: " + constr[0] + " " + constr[1])

7. Get an individual

>>> print ( optjson.getIndividual(0) )

8. Get an individual with specific ID

>>> print ( optjson.getIndividualWithID(18) )

raise error

>>> except:
    >>> print ( sys.exc_info()[1] )

__actual_parse(filename)

Parse an optimizer JSON file

Parameters:

filename (str) – JSON file to be loaded

__clear_container()

Clear old attribute data

__hasLoaded()

Throw error if no generation or pareto front file loaded.

__init__()

Constructor.

__parse_version_2_0_0(data)

Read in data

Parameters:

data (str) – A generation file

Notes

Storage of columns: dvars, objs, ID

__parse_version_2_1_0(data)

Read in data

Parameters:

data (str) – A generation file

__reset_attributes()

Reset attributes to default values.

check_file(filename)

Check if a file is really an optimizer output

Parameters:

filename (str) – JSON file to be loaded

Returns:

True if an optimizer file, otherwise False

Return type:

bool

clear()

Clear data.

getAllInput()

Obtain all design variable input.

Returns:

A ndarray where each column corresponds to the values of a design variable

Return type:

numpy.ndarray

getAllOutput()

Obtain all objective output.

Returns:

A ndarray where each column corresponds to the values of an objective

Return type:

numpy.ndarray

getBounds(dvar='')

Obtain the design variable bounds

Parameters:

dvar (str, optional) – Design variable or ‘’

Returns:

If dvar == ‘’ it returns the whole dictionary of design variables and their bounds otherwise it returns the bounds of the given design variable

Return type:

dict or array

Notes

Raises an exception if design variable unknown

getConstraints()

Obtain the constraints of the simulation. They are formulas written as strings

Returns:

A list of strings containing the constraints

Return type:

list [str]

getDesignVariables()

Obtain names of all design variables

Returns:

A list containing all design variable names

Return type:

list

getIDs()

Obtain all individual IDs.

Returns:

A list of IDs

Return type:

list [int]

getIndexOfID(ID)

Get Index of ID

Returns:

The index of the individual with certain ID

Return type:

int

getIndividual(ind)

Get individual in a list of [dvars, objs, ID]

Parameters:

ind (int) – The individual number (not the ID)

Returns:

A list containing [design variables, objectives, ID]

Return type:

list

Notes

Raises an index error if individual number is out of bounds

getIndividualWithID(ID)

Get invdividual with ID

Parameters:

ID (int) – ID for which we want the data

Returns:

A list containing [design variables, objectives, ID]

Return type:

list

Notes

Raises a runtime error if there’s no individual with given ID

getNumOfGenerations()

getObjectives()

Obtain names of all objectives

Returns:

A list containing all design variable names

Return type:

list

property num_optimizers

parse(directory)

Collect information about optimizer JSON files.

Parameters:

directory (str) – Where the .json files are

Notes

Raises a runtime error if

• there are several several different basenames

• no json file is found

readGeneration(gen, opt=0, pareto=False)

Parse a generation file

Parameters:

• gen (int) – The generation number

• opt (int, optional) – The optimizer number (default: 0)

• pareto (str, optional) – Read a pareto file (default: False)

Notes

Raises a runtime error if generation file doesn’t exist

opal.parser.PeakParser module

class opal.parser.PeakParser.PeakParser

Bases: BaseParser

Read measurement file of the form:

# Peak Radii (mm), 18 June 2008, 200 uA (RRI2-65)
2103.8
2122.4
2140.4
2159.8
2180.8
2201.8
2224.6
2247.8
2270.0
2287.8
2302.6
2315.6
2331.4
2356.0
2379.6
2403.6
2424.6

It is also able to readd simulation files of the same form.

_peaks

Dictionary with peak file info =

‘type’

Simulation or measurement

‘name’

List of all supported variables

‘dataset’

List of radii and intensity

‘unit’

List of units

Type:

dict

__init__()

clear()

Clear data.

getDataOfVariable(var)

getType()

getUnitOfVariable(var)

getVariableName()

isVariable(var)

parse(filename)

opal.parser.SDDSParser module

class opal.parser.SDDSParser.SDDSParser

Bases: BaseParser

__removeNumber(s)

_checkVersion(filename)

_column(f)

_data(f)

_description(f)

_get_type(dtype)

_hasVariable(varname)

_parameter(f)

_parseHeader1_6(filename)

_parseHeader1_9(filename)

_version(v)

clear()

Clear data.

collectStatFileData(baseFN, root, yNames)

FIXME: This function shouldn’t be part of the parser.

Assumes runOPAL structure: optLinac_40nC_IBF=485.9269768907996_IM … where baseFN == optLinac_40nC in the example above.

This function finds all stat files that are one level below root. An exclude list can be specified.

Two vectors are returned: x with the design variables names and values (IBF=485.9269768907996 … from above) and the last value(s) of stat file data spcified via yNames.

Example: x = [] y = [] baseFN = ‘optLinac_40nC’ exclList = [‘tmpl’,’extrData.py’] root = “.” yNames = [‘s’,’energy’] p = SDDSParser() (x,y) = p.makeData(baseFN, exclList, root, yNames)

property dataframe

getDataOfVariable(varname)

getDescriptionOfVariable(varname)

getUnitOfVariable(varname)

getVariables()

parse(filename)

property size

opal.parser.TimingParser module

class opal.parser.TimingParser.TimingParser

Bases: BaseParser

Read and write an Ippl timing file.

_data

List of dictionaries

Type:

list

Notes

It stores the data in a list of dictionary where the main timing dictionary is:

main_dict = {'cpu tot':     [],
             'wall tot':    [],
             'what':        [],
             'cores':       []}

and the specialized timings are in stored in a dictionary with the structure:

special_dict = {'what':     [],
                'cpu max':  [],
                'wall max': [],
                'cpu min':  [],
                'wall min': [],
                'cpu avg':  [],
                'wall avg': []}

The list of dictionaries looks then as follows:

self._data = [main_dict,
              special_dict_1,
              ...,
              special_dict_N]

Examples

>>> TimingParser as timing
>>> time = timing.TimingParser()
>>> time.read_ippl_timing("/path/to/IpplTiming.dat")
>>> data = time.getTiming()
>>> print ( data )
>>> time.pie_plot('cpu avg'):

__init__()

_exportAscii(pathname, data)

Write a human readable file

Parameters:

• pathname (str) – Path + filename of written file

• data (list [dict]) – Timing data

Notes

If pathname has no extension the string “.dat” is appended

_exportPickle(pathname, data)

Write a binary pickle file

Parameters:

• pathname (str) – Path + filename of written file

• data (list [dict]) – Timing data

Notes

If pathname has no extension the string “.pkl” is appended

_init_data_structure()

This is the way the timing is stored

Returns:

• main_dict (dict) – The dictionary for the main timer

• special_dict (dict) – The dictionary for all other timers

_load_pkl(pkl_file)

Pickle file loading function

Parameters:

pkl_file (str) – Pickle timing file to load

_order(line, i)

Find the order of the tags, i.e. ‘cpu min’, etc. and fill dictionary.

Parameters:

• line (str) – File line

• i (int) – Start index

Returns:

A dictionary giving tag as key and occurrence as number.

Return type:

dict

clear()

Clear data.

getProblemSize()

Returns:

All problem specification in a dictionary

Return type:

dict

Notes

It is not checked if the container is empty.

getTiming()

Returns:

The timing data

Return type:

list

Notes

It is not checked if the container is empty.

parse(filename)

property properties

read(pathname, info=False)

Parameters:

• pathname (str) – Path + filename of pickle file

• info (bool, optional) – Print data when reading

read_ippl_timing(f)

Read in an Ippl timing file.

File created by:

std::string filename = "myTiming.dat";
Ippl:print(filename, problemSize);

The problem size is optional.

Parameters:

f (str) – Pathname (i.e. path + filename)

read_output_file(f)

Read in the timing results from an OPAL output file.

Parameters:

f (str) – The pathname (i.e. path + filename)

Notes

Following format assumed:

Timings{0}> -----------------------------------------------------------------
Timings{0}>      Timing results for 32 nodes:
Timings{0}> -----------------------------------------------------------------
Timings{0}> mainTimer........... Wall tot =    326.192, CPU tot =     325.76
Timings{0}>
Timings{0}> my awesome timer.... Wall max =          0, CPU max =          0
Timings{0}>                      Wall avg =          0, CPU avg =          0
Timings{0}>                      Wall min =          0, CPU min =          0
Timings{0}>
Timings{0}> super timer......... Wall max =    14.6091, CPU max =      14.52
Timings{0}>                      Wall avg =    3.34291, CPU avg =    3.31844
Timings{0}>                      Wall min =   0.007039, CPU min =          0
Timings{0}>
Timings{0}> best timer.......... Wall max =    33.4165, CPU max =      32.93
Timings{0}>                      Wall avg =    23.0727, CPU avg =    22.8328
Timings{0}>                      Wall min =     19.989, CPU min =      19.67
Timings{0}>
Timings{0}> -----------------------------------------------------------------

write(pathname, form='PICKLE', data=None)

Export a timing data in a specific format

Parameters:

• pathname (str) – Path + name of the written file

• data (list [dict], optional) – Timing data

• form (str, optional) – Which format to write

Notes

Throws an exception if the format is unknown or not available

opal.parser.TrackOrbitParser module

class opal.parser.TrackOrbitParser.TrackOrbitParser

Bases: BaseParser

__init__()

clear()

Clear data.

property description

getDataOfVariable(var)

getUnitOfVariable(var)

getVariableNames()

isVariable(var)

parse(filename)

property size

opal.parser.sampler module

class opal.parser.sampler.SamplerParser

Bases: BaseParser

Parses *.json files of OPAL that are written by the SAMPLE command.

Notes:

Supports following JSON formats:

{
    "name": "sampler",
    "dvar-bounds": {
        "MX": "[ 16, 32 ]",
        "nstep": "[ 10, 40 ]"
    },
    "samples": [
        {
            "ID": "0",
            "dvar": {
                "MX": "20",
                "nstep": "33"
            }
        },
        {
            "ID": "1",
            "dvar": {
                "MX": "21",
                "nstep": "29"
            }
        }
    ]
}

{
    "samples": {
        "0": {
            "dvar": {
                "MX": "16",
                "nstep": "10"
            },
            "obj": {
                "o1": 22.2,
                "o2": 22.1
            }
        },
        "1": {
            "dvar": {
                "MX": "19",
                "nstep": "11"
            },
            "obj": {
                "o1": 21.2,
                "o2": 23.1
            }
        },
        "2": {
            "dvar": {
                "MX": "22",
                "nstep": "12"
            },
            "obj": {
                "o1": 12.2,
                "o2": 32.1
            }
        }
    },
    "name": "sampler",
    "OPAL version": "2.0.0",
    "git revision": "1849b7e5130657e8be50d524de0f6c50134f330a",
    "dvar-bounds": {
        "MX": "[ 16, 32 ]",
        "nstep": "[ 10, 40 ]"
    }
}

__id

(= ‘sampler’) used to identify the file to be a SAMPLE output

Type:

str

__tag

(= ‘name’) id tag for file identification

Type:

str

__begin

Start individual id

Type:

int

__end

End individual id

Type:

int

__dvars

The design variables of each individual

Type:

list

__dvar_bounds

All design variable bounds

Type:

dict

__objs

The objectives of each individual

Type:

list

__fix_bound_type()

Fixes type of DVAR bounds. In the JSON file the bounds are in a string, e.g. ‘[0, 1]’.

We need to change to list of floats.

__init__()

__parse_version_2_0_0(data)

__parse_version_2_1_0(data)

property begin

returns: Lowest individual ID :rtype: int

property bounds

Obtain design variable upper and lower bounds

Returns:

A dictionary of design variable names (key) and their bounds

Return type:

dict

clear()

Clear data.

property design_variables

Obtain names of design variables.

Returns:

A list of strings

Return type:

list [str]

property end

returns: Highest individual ID :rtype: int

getIndividual(ind)

Obtain input values of an individual

Parameters:

ind (int) – Individual number

Returns:

A dictionary with design variable names (keys) and their input value.

Return type:

dict

getObjectives(ind)

Obtain output values of an individual

Parameters:

ind (int) – Individual number

Returns:

A dictionary with design variable names (keys) and their input value.

Return type:

dict

property num_samples

property objectives

Obtain names of objectives.

Returns:

A list of strings

Return type:

list [str]

parse(filename)

Load the JSON file.

Parameters:

filename (str) – JSON file to be loaded