OSGeo Planet

gvSIG Team: Towards gvSIG 2.5: New geoprocess, Ring map

OSGeo Planet - Fri, 2019-02-22 14:13

In the last gvSIG Conferences we talked about a series of geoprocess created for criminology developed with Universidad Jaume I (UJI). You can see the video here (in spanish, if you have any question you can write directly to us and we will help you). This geoprocess can be used in different fields, not only in criminology.

The first geoprocess that we are going to present is the Ring Map. The objective of this geoprocess is to make easy to visualize info that is inside the layer. For example, it will show info in multiple  fields in a single map.

In this example, we are going to see a layer with the population on each block for different range of ages. The result of the ring map will be like in the image. Now with this graph, with a quick look, we can analyse quickly how the population is distributed in each block and in different age ranges.

This geoprocess generate a series of rings and sectors. Each ring is a reference for the same field, and each sector is for the same geometry, that is pointed with an arrow. It creates three new layers with the rings, labels and arrows. Each of them can be modified as a normal shapefile. It creates an automatic legend that can also be modified as we need them.

The parameters that we need to execute this geoprocess is, at least one layer, (with any type of geometry) that it has a field with unique values for each geometry. It needed a opened table. If you are going to use layers,  be sure that you open previously the attributes table, if not, this  table will not appear as an option. If it doesn’t have an unique field, you can create it using the Field Calculator. This option has been created  becouse it is posible to create diferent representations (differents table) to show them over the same vectorial layer.

The fields should be written in the order that we want them to appear. Also, we have multiple options for change the visualization of the rings. We can change sectors, gaps, size, labels..

This is a similar result from the initial image.

As always, you can see more information about this process pressing the help button.

You can test the Ring Map geoprocess  at the new distribution candidate to final version. Just go to Tools -> Addons Manager and Search for “Geoprocess: Ring Map”.

If you find any error we encourage you to write about it at the users mailing list. It will help us to fix them.

Categories: OSGeo Planet

gvSIG Team: Camino a gvSIG 2.5: Nuevo Geoproceso, Mapa de anillos

OSGeo Planet - Fri, 2019-02-22 13:45

Ya comentamos en las últimas jornadas de gvSIG una serie de geoprocesos relacionados con criminología junto a la Universidad Jaume I (UJI). Puedes ver el video de la ponencia aquí. Estos geoprocesos pueden utilizarse en diferentes campos, no ligado en exclusiva a la criminología.

El primer geoproceso  que vamos a presentar es el de Mapa de anillos. El objetivo es el de facilitar la visualización de una capa que dispone de información en diferentes campos y necesitamos mejorar la visualización de todos ellos para presentarlos en un mapa.

En el siguiente ejemplo, vamos a ver una capa que contiene la población en diferentes campos de la tabla de atributos, para visualizarlo generaremos un mapa de anillos que dará como resultado algo similar a la imagen de a continuación. De esta forma podemos ver de un vistazo rápido como se distribuye la población en diferentes franjas de edades sobre una zona del mapa.

El geoproceso genera una serie de anillos y sectores. Cada anillo hace referencia a un mismo campo y cada sector hace referencia a una misma entidad o geometría, con una flecha que señala la geometría de la cual extrae los valores. Vemos que genera una serie de capas sobre la vista que podemos tratarlas como cualquier otra capa de gvSIG. Dispone de una leyenda que se genera de forma automática y de una serie de capas de puntos que sirven para colocar las etiquetas. Todas ellas son modificables para adaptarse  a las necesidades que tengamos.

Los parámetros necesarios para ejecutar este geoproceso es mínimo una capa (puede ser de cualquier tipo de geometría) que tenga de forma obligatoria un campo con un valor único que identifique cada geometría. Tiene que haber una tabla abierta (si usas solo capas vectoriales asegúrate de abrir la tabla de atributos con antelación para poderla seleccionar sino no aparecerá en el listado de tablas disponibles). Si no lo tuviera podríamos crearle uno desde la calculadora de campos. Este campo único tiene el uso de que sobre una misma capa vectorial, podríamos tener diferentes resultados a mostrar en diferentes tablas cuyos valores podrían ser calculados en otros programas. El geoproceso relacionará esos valores por su valor único y los mostrará.

Los campos a mostrar deberán ser escritos por orden en que queremos que se muestren. También disponemos de muchas opciones de configuración para la creación de los anillos. Desde posición de los sectores, espacio entre ellos, tamaño del circulo interior, del grosor de cada sector, creación de etiquetas..

Aquí puedes ver otra representación similar a la de la imagen inicial.

Siempre puedes ver con más detalle la información del geoproceso presionando el botón de ayuda.

Ya puedes descargarte y probar el nuevo geoproceso de mapa de anillos en la nueva candidata a versión final . Puedes descargarlos de Herramientas -> Administrador de complementos y buscar por el geoproceso “Geoprocess: Ring Map”

Si encuentras cualquier fallo o recomendación te animamos a escribirnos a las Listas de correo de gvSIG.

Categories: OSGeo Planet

Paul Ramsey: Proj 6 in PostGIS

OSGeo Planet - Fri, 2019-02-22 13:00

Map projection is a core feature of any spatial database, taking coordinates from one coordinate system and converting them to another, and PostGIS has depended on the Proj library for coordinate reprojection support for many years.

Proj 6 in PostGIS

For most of those years, the Proj library has been extremely slow moving. New projection systems might be added from time to time, and some bugs fixed, but in general it was easy to ignore. How slow was development? So slow that the version number migrated into the name, and everyone just called it “Proj4”.

No more.

Starting a couple years ago, new developers started migrating into the project, and the pace of development picked up. Proj 5 in 2018 dramatically improved the plumbing in the difficult area of geodetic transformation, and promised to begin changing the API. Only a year later, here is Proj 6, with yet more huge infrastructural improvements, and the new API.

Some of this new work was funded via the GDALBarn project, so thanks go out to those sponsors who invested in this incredibly foundational library and GDAL maintainer Even Roualt.

For PostGIS that means we have to accomodate ourselves to the new API. Doing so not only makes it easier to track future releases, but gains us access to the fancy new plumbing in Proj.

Proj 6 in PostGIS

For example, Proj 6 provides:

Late-binding coordinate operation capabilities, that takes metadata such as area of use and accuracy into account… This can avoid in a number of situations the past requirement of using WGS84 as a pivot system, which could cause unneeded accuracy loss.

Or, put another way: more accurate results for reprojections that involve datum shifts.

Here’s a simple example, converting from an old NAD27/NGVD29 3D coordinate with height in feet, to a new NAD83/NAVD88 coordinate with height in metres.

SELECT ST_Astext( ST_Transform( ST_SetSRID(geometry('POINT(-100 40 100)'),7406), 5500));

Note that the height in NGVD29 is 100 feet, if converted directly to meters, it would be 30.48 metres. The transformed point is:

POINT Z (-100.0004058 40.000005894 30.748549546)

Hey look! The elevation is slightly higher! That’s because in addition to being run through a horizontal NAD27/NAD83 grid shift, the point has also been run through a vertical shift grid as well. The result is a more correct interpretation of the old height measurement in the new vertical system.

Astute PostGIS users will have long noted that PostGIS contains three sources of truth for coordinate references systems (CRS).

Within the spatial_ref_sys table there are columns:

  • The authname, authsrid that can be used, if you have an authority database, to lookup an authsrid and get a CRS. Well, Proj 6 now ships with such a database. So there’s one source of truth.
  • The srtext, a string representation of a CRS, in a standard ISO format. That’s two sources.
  • The proj4text, the old Proj string for the CRS. Until Proj 6, this was the only form of definition that the Proj library could consume, and hence the only source of truth that mattered to PostGIS. Now, it’s a third source of truth.

Knowing this, when you ask PostGIS to transform to an SRID, what will it do?

  • If there are non-NULL values in authname and authsrid ask Proj to return a CRS based on those entries.
  • If Proj fails, and there is a non-NULL srtext ask Proj to build a CRS using that text.
  • If Proj still fails, and there is a non-NULL proj4text ask Proj to build a CRS using that text.

In general, the best transforms will come by having Proj look-up the CRS in its own database, because then it can apply all the power of “late binding” to ensure the best transformation for each geometry. Hence we bias in favour of Proj lookups, then the quite detailed WKT format, and finally the old Proj format.

Categories: OSGeo Planet

Cameron Shorter: Inspiring techies to become great writers

OSGeo Planet - Fri, 2019-02-22 07:13

How to attract writers into Open Source communities, have them amplify developers' written impact, reduce tech ramp-up, facilitate growth, while having fun improving the world.

Reading time: 8 minutesPresented at Write The Docs - Sydney [slides]
My story starts at the international conference for Free and Open Source Software for Geospatial when it was held here in Sydney, ten years ago. We wanted to build a distribution of the software to hand out to conference attendees. Back then when software was expensive to download, it was a useful handout, and it provided an attractive marketing pipeline for these free software projects.To build the first version of OSGeoLive we followed install instructions, configured applications, resolved dependencies and conflicts, asked questions on email lists, and finally, after much hard work, produced our first release. This was a very important first step as it showed that we had the commitment to follow through and deliver.
However, it also highlighted many of our failings. We had followed a fully manual install process, so when the next software releases came out, we had to start the installation process again, from scratch. It was completely unsustainable. We needed to automate the install process, and we needed projects to show us how.
But our first calls for help were embarrassingly underwhelming. You see, the perceived learning curve for packaging was considered unacceptably high and volunteers were not stepping up.
To fix this, we wrote an example install script along with clear, step-by-step instructions. We then went back to developers with the message of "if you can spend a couple of hours writing a short install script which looks like this, then we will market your application though our OSGeoLive distribution". Small effort / High Value. And it worked! 28 projects packaged their applications for our version 2 release.In following releases, we applied the same process to documentation. Our first docs were written by developers and as such were quite ... variable.  So we created skeleton outlines for Project Overviews and Quickstarts, along with clear writing instructions. We then asked projects to write documentation in line with theses outlines. We followed up with reviews, and a publishing pipeline, to produce consistent, quality material. So by applying an outline / write / review / publish process, we were able to achieve significant efficiencies by allowing experts to focus only on the bits they do best and thus reduce contribution effort.
Translators later adopted the same formula to create multiple language variants.
Unpacking the storyOk, let’s unpack this story from the perspective of technical writers. Firstly, we should acknowledge the significant efficiencies software enables. Four out of the ten richest people in the world are self made software entrepreneurs.
However, while proprietary businesses centralise captured wealth, open source software, like other creative commons works, is given away for free. This act of sharing has far reaching consequences. Tools and knowledge created can be used and built upon by everyone. This democratises knowledge, wealth and power.
So if you want to positively improve our world, supporting Open Source projects is a good place to start.
The ResearchSo what makes a successful Open Source project; one which is sustained in the long term? Research provides some answers,  An analysis of over 170,000 Open Source projects to identify success factors, and a study of the success factors of episodic volunteering in Open Source both highlight the importance of:
  • Clear utility,
  • Lean governance,
  • Clear vision and goals,
  • Marketing and a quality website,
  • Good user and developer documentation,
  • Digestible information,
  • Guided introductions,
  • Simple workspaces,
  • Task–finding dashboards,
  • Financial backing,
  • Fine scaled task granularity, making it easier for people to contribute.
And on the social side:
  • Leaders who lead by doing, by putting in the hard work,
  • A virtuous and supportive community,
  • Attracting external contributors,
  • Advocating broadly, by all involved,
  • A sense of collective ownership,
  • Hosting of meetups,
  • Recognising all forms of contribution,
  • Personal invitations for help,
  • Showing appreciation.
Notice the prevalence of communication within these characteristics?Communicators Needed
    Where I’m heading here is that while the Open Source story usually focuses on software developers, equally important are communicators and community builders. So for you folks who are technical writers, Open Source communities need you! Your involvement can significantly amplify a project’s impact, and by extension, your help will be:
    • Making the world more egalitarian, 
    • Reducing wealth inequality, 
    • And democratising knowledge and power. 
    Bridging the tech gapI’d like to touch on inflection points. As a project’s technology and user base matures, community success factors move. An early inflection point with Open Source projects happens when a project attracts external contributors. These contributors increase the size of the workforce and adds extra skills, but they lack the deep domain expertise of the core team. To attract contributors, projects should to reduce ramp up time; and good documentation is key. Poetically however, the documenters who are most qualified to help are typically the least technical, and face the greatest ramp up time.To help explain this problem, I’d like to introduce you to Johanna Botman. She started her working career as an english teacher, then moved into IT, website design, and is now a geospatial officer in local council. I met her at a spatial conference’s community day when she joined other volunteers in updating our OSGeoLive documentation. It was a humbling experience for me. I saw such an accomplished person, with oodles of commitment, struggling to use our documentation tools.Johanna reviewed an early draft of this presentation and observed that ...Some of the issues relating to getting started is about bridging the gap between developers and writers.  Developers write code in coding tools. They collaborate, they are used to versioning and are at home with unformatted raw text and automation tools. Writers? They work on Windows machines in Word – maybe in Google Docs if they are lucky. They don’t know about running build scripts, running Linux variants, Github, chat programs, HTML, RST, Wikis and the variants of markup languages.
    It’s one thing to work with the Open Source software, another to write about it, and a third thing to work out how to write it so that it fits in with the project. It seems as if the developers have created the writing system in a way that they are used to working, not necessarily in a way that works for writers.And I think Johanna is right. I’ve found many users and writers who are keen to help, but feel overwhelmed, don’t know where to start, don’t want to burden the existing community, and so are shy to ask questions.Writers need to be sought out, welcomed, encouraged, supported and publicly appreciated if projects are to attract good documenters to help them grow.In turn, I’d encourage writers to proactively reach out to development communities. I’d expect you’ll find them very welcoming and appreciative, especially if you are volunteering to tackle key documentation pain points.And together we need to help bridge the gap between writers and developers.Documentation StrategySo what should writers expect to see when joining a project. Typically, emerging projects have patchy documentation, in various stages of currency, completion, relevance, verbosity, and quality. The challenge is to consolidate it so that it becomes concise, intuitive, accurate, minimises learning and ramp up time, and sustainably maintained to match continually evolving software releases. This is non-trivial and requires good writing skills, planning, leadership, commitment and follow through to do it well.The problems are that:
    • Software is complex. It’s hard to understand, and harder to explain.
    • Software continually evolves, meaning documentation needs to be continually maintained.
    • Projects have a range of audiences, with differing information needs, technical backgrounds, and attention spans.
    It helps to define the characteristics of your different communication mediums.
      Raw image (in draw.io format)Your front web page and elevator pitch should be concise, highly polished and target first time visitors, while community forums are good for once off questions and can be rough around the edges.For each medium you should define your:
      • Target audience,
      • Document structure,
      • Technical depth,
      • Quality requirements, and
      • Maintenance strategy.
      Then aim to:
      • Write timeless material to minimise maintenance.
      • Only write as much as your team has capacity to maintain.
      • Help techies become great writers. Provide skeleton docs and writing instructions, then review, mentor and publish.
      • Work within a release cycle, which ideally synchronises with the project. 
      It’s worth noting that most of what I’m talking about here is relevant to all sorts of technical writing - not just for Open Source communities.
      Getting Started?Okay, so maybe you’d like to get involved and are wondering where to start? I’d suggest:Image source
      • Focus on one project instead of many, as your ramp-up time will likely be high. 
      • Acknowledge the value you bring. Use it as motivation because you’ll have many challenges and the inner motivation will really help.
      • Demonstrate commitment. It’s contagious.
      • Ask for help, you will need it, and you’ll likely be surprised how supportive people will be.
      • Don’t be daunted by your lack of domain knowledge. It’s actually an asset when writing to a broad audience.
      • Help developers become great writers. Great impact comes from amplifying the effectiveness others. 
      • Be part of the team; help paint the vision; be inspirational.
      • And most of all, have fun while you are doing it. Because believe you me, it is hugely rewarding to share the team camaraderie involved in building something that is much bigger and more impactful than you could possibly create by yourself.
      Categories: OSGeo Planet

      Martin Davis: Better/Faster Interior Point for Polygons - now in GEOS

      OSGeo Planet - Thu, 2019-02-21 19:52
      As a gentle introduction to GEOS development I took on the task of porting the recent improvements to the JTS InteriorPointArea algorithm.   This is a fairly small chunk o'code, but it touches most of the aspects of GEOS development process: build chain, debugging, unit tests, and infra (source control and build farm).  It also has the advantage that the JTS code was still hot off the keyboard and (almost) unreleased, so it was a chance to see if any cross-fertilization would blossom from working on the two projects jointly.

      Skipping lightly over the details of my (somewhat painful) GEOS learning curve, I'm delighted to say that the code has landed in master and is basking in the green glow from the build bot badges.

      And now the whole point of the exercise: how much better is the new code in GEOS? 

      It exhibits the expected improvement in robustness, since a GEOS test which actually depended on a thrown TopologyException (due to the now-removed call to Geometry::intersection() ) had to be modified to handle a successful return.

      Most importantly, there is a dramatic improvement in performance.  Here's some numbers from running the GEOS InteriorPointArea performance test:

      Data sizeTimeTime
      Centroid100.8 ms86 msx 1001 ms10006 ms144 msx 2412 ms10,00055 ms672 msx 12107 ms100,000508 ms6,714 msx 13961 ms1,000,0005,143 ms73,737 msx 1411,162 ms
      Some observations:
      • The performance test uses synthetic data (sine stars).  Real-world datasets are likely to show significantly better times ( 80x better in some cases, based on JTS timings)
      • The largest improvement is for small geometries, which is nice since these are more common
      • InteriorPoint is now actually faster than the Centroid computation.  This is also good news, since users were often tempted to try and use centroids instead of interior points, despite the known issues.
      Future Work
      Running the identical performance test in JTS is still faster, by roughly 5x.  This may be due to the advantages of JIT compilation and memory management.  It may also indicate there is room for improvement by making GEOS smarter about data handling.
      Categories: OSGeo Planet

      gvSIG Team: Towards gvSIG 2.5: Improvements at geoprocessing framework

      OSGeo Planet - Thu, 2019-02-21 13:00

      The gvSIG Desktop 2.5 version, that you can test with the new distribution candidate to final version, includes a series of improvements in the geoprocess framework that we explain below. 

      Visual improvements

      Now when opening the geoprocessing toolbox, you can see that extra information about the layers and fields appears at the interface.

      At layer type parameters, you will be able to see the layer name, the name of the View in which it is, if it has an active selection and how many entities are selected. An icon will also appear indicating the type of geometry of the layer, or if it has a selection (with a background yellow circle).

      At field type parameters, you can also see information about the type of field. and in the future you will be able to see an icon identifying that field.

      Integration with the new plugin to capture coordinates

      Last year we released a new extension to capture coordinates more effectively in the View. At the new version, the integration with the geoprocess framework has been completed, so that the points captured with that tool will appear at the selection of the Point type parameter.

      New Filter type parameter

      As we have told about and as we will explain in the next posts in detail, gvSIG has a new expression generator. In the geoprocessing framework this functionality has been added, so that new geoprocesses can have a new type of parameter that is expression type.

      This parameter can greatly improve the execution of a geoprocess by a filter and it is also much more optimal to use than a selection done with the ‘Selection by attribute’ tool. It can greatly improve the processing of some geoprocesses.

      This parameter, according to the geoprocess, could be used for values calculation tasks, filtering, etc.

      The expression type parameter is associated to each layer or table type parameter, so that several of these parameters may exist in the same geoprocess. Each time we run the expression generator we will see that this window is associated with the corresponding initial parameter. Now you can start to test all these improvements at the new distribution candidate to final version and if you find any error we encourage you to write about it at the users mailing list. It will help us to fix them.


      Categories: OSGeo Planet

      gvSIG Team: Camino a gvSIG 2.5: Mejoras en el marco de geoprocesos

      OSGeo Planet - Wed, 2019-02-20 13:38

      La versión de gvSIG Desktop 2.5, que ya puedes testear con la nueva candidata a versión final, trae una serie de mejoras en el marco de geoprocesos que explicamos a continuación.

      Mejoras visuales

      Ahora al abrir los geoprocesos, podremos notar que en la interfaz nos aparece información extra sobre las capas y los campos que antes no teníamos.

      En los parámetros de tipo capa, podremos ver rápidamente el nombre de la capa, el nombre de la Vista en la que se encuentra, si tiene una selección activa y de cuantas entidades es esa selección. También aparecerá un icono que nos indica el tipo de geometría de la capa o si tiene una selección (con un circulo amarillo de fondo).

      En los parámetros de tipo campo, también podremos ver información sobre el tipo de campo que es y a futuro veremos un icono identificativo de dicho campo.

      Integración con el nuevo capturador de coordenadas

      El año pasado sacamos una nueva extensión para capturar coordenadas de manera más efectiva en la Vista. Con la nueva versión, se ha completado la integración con el marco de geoprocesos, por lo que los puntos capturados en la herramienta, aparecerán en la selección del parámetro de tipo Punto.

      Nuevo parámetro de tipo Filtro

      Como hemos comentado y explicaremos en próximos post con más detenimiento, gvSIG tiene un nuevo generador de expresiones. En el marco de geoprocesos hemos añadido esta funcionalidad para que nuevos geoprocesos puedan tener un nuevo tipo de parámetro que sea de tipo expresión.

      Este parámetro puede agilizar enormemente la ejecución de un geoproceso por un filtro y además es mucho más optimo de utilizar que una selección realizada con el Selección por atributos, lo que puede mejorar enormemente la velocidad de algunos geoprocesos.

      Este parámetro, según el geoproceso, podría utilizarse para tareas de cálculo de valores, de filtrado, etc.

      El parámetro de tipo expresión, va asociado a cada parámetro de tipo capa o tabla, por lo que en un mismo geoproceso pueden existir varios de estos parámetros. Cada vez que le demos al generador de expresiones veremos que esa ventana está asociada al parámetro inicial que corresponde.

      Ya puedes comenzar a probar todas estas mejoras en la nueva candidata a versión final y si encuentras cualquier fallo o recomendación te animamos a escribirnos a las Listas de correo de gvSIG.


      Categories: OSGeo Planet

      Just van den Broecke: Emit #6 – AirSensEUR Calibration

      OSGeo Planet - Tue, 2019-02-19 16:23

      This is Emit #6, in a series of blog-posts around the Smart Emission Platform, an Open Source software component framework that facilitates the acquisition, processing and (OGC web-API) unlocking of spatiotemporal sensor-data, mainly for Air Quality and other environmental sensor-data like noise.

      In Emit #5 – Assembling and Deploying 5 AirSensEURs…, I described how , with the great help of Jan Vonk from RIVM, we placed five AirSensEUR (ASE) air quality stations at the RIVM reference site near the A2 Highway at Breukelen. For about 2.5 months raw data was gathered there while the RIVM station was gathering its data to be used as reference for calibration.

      Now “calibration” is a huge and increasingly important topic when using inexpensive sensors for measuring Air Quality. Within the Smart Emission project we have been applying Artificial Neural Networks to calibrate the gas-sensors within the Josene stations. See also the SE documentation. These sensors were so called metaloxide (MICS) sensors from SGX Sensortech Limited.

      The AirSensEURs contain electrochemical sensors from AlphaSense. Several sources like RIVM, state that these sensors are more accurate (than metaloxide sensors), but at the same time need per-sensor calibration.

      Within the ASE boxes the following four gas-sensors were applied:

      The Gang of Four Sensors

      The calibration to be applied is based on Regression Analysis. This and other calibration-methods have been investigated and evaluated for many types/brands of sensors by the EC-JRC team. Read all in this landmark article and other references there. 

      The complete timeline was as follows. Each phase will be expanded below.

      1. Aug 1, 2018 – Okt 9, 2018
        Raw ASE and RIVM reference data collection (Breukelen site)

      2. Okt 10, 2018 – Nov 2, 2018
        All ASEs deployed at their target locations

      3. Nov/dec 2018
        Calibration performed by Michel Gerboles at the EC-JRC lab 

      4. Jan 2019
        Calibration formulas implemented in Smart Emission (SE) platform

      5. Feb 2019
        All ASE calibrated gas-data continuously available via SE viewers/APIs

      6. Feb 2019
        Analysis of the calibration (SE Python Stetl) implementation results

      ad 1) The five ASE Boxes were mounted on a horizontal pole and connected to WIFI and current. As end-result all boxes were publishing their raw data to the SE InfluxDB Data Collector and were visible in the SE Grafana raw data viewer.

      Configured for InfluxDB Data Push visualized via Grafana

      Reference values for the gases NO,NO2 and O3 were already continuously available from the RIVM reference station via e.g. the RIVM SOS API. A CO-sensor was added to the station by RIVM. CO-data was collected manually.

      ad 2) The picture below shows ASE_NL_01 (left above) through _05 clockwise at their deployment sites. ASE_03 and 04 (right below) were at a single location.

      ASE_NL 01 was deployed at an RIVM site in Nijmegen. This allowed us to verify its calibration with different reference data as with which it was calibrated! See below.

      ad 3) The calibration was performed by EC-JRC (M. Gerboles) using R and ShinyR webapp. All sources can be found in this EC-JRC GitHub repo. This process is quite intricate and a bit hard to explain in the context of a blog-post paragraph. I’ll try a summary:

      Sensor values are digital readings (0..65535). This is effected by the electrical circuitry within each ASE, for optimal gain. To calculate back to mV and nA a per-sensor (brand+type) calculation is required first before applying any regression formula. A bit is explained in the image below.

      The second outcome is a per-individual-sensor regression formula. This is for most sensors a linear equation. For O3 (OX_A431) the formula is polynomial, as O3 readings are influenced by NO2 concentration. Below is an example as later implemented in Python using SE Stetl ETL (see below).

      The main three outcomes of the calibration are:

      • the parameters for digital to nA calculation (per sensor brand+type)
      • the linear (polynomial) equations for nA to concentration (ug/m3)
      • the per-individual-sensor parameters (a0-a3)

      Above some scatterplots made for ASE Box 3 NO2 and O3.

      ad 4). Knowing all equations and their parameters from step 3 above, I attempted to integrate this in the continuous ETL within the Smart Emission Platform. Up to now the platform supported only a single sensor station type: the Intemo Jose(ne). As the platform is fed by harvesting raw data from a set of remote APIs provided by Data Collectors, it was relatively easy to add sensor(-station)-metadata and extend the Refiner ETL to apply calibration algorithms driven by that metadata. 

      So for Josene stations the existing ANN calibration would still be applied, while for ASE stations per-sensor linear equations would be performed. All parameterization was already configurable using the Device, DeviceDefs, DeviceFuncs abstractions in the SE Stetl implementation. Recently, to allow stations that already send calibrated values, I introduced the Vanilla Device starting with harvesting Luftdaten.info stations (more in a later post).

      The formula’s as applied in Python SE Stetl are as follows:

      STEP 1a - Digital to Voltage (V)
      V = (Ref - RefAD) + (Digital+1) /2^16 x 2 x RefAD STEP 1b - Voltage (V) to Ampere (I) as Ri
      I = 10^9 V/(Gain x Rload) STEP 2 - Ampere (I) to concentration (ug/m3) - Example
      I=a0+a1*NO2+a2*T ==> NO2 = (I - a0 - a2 * T) / a1
      a0-a2 has specific values for each NO2-B43F sensor.

      Now that these formulas and their parameters were implemented, near-realtime values could be made visible in all SE apps (viewers) and APIs such as the SmartApp and the Heron Viewer.

      Within the Heron Viewer we can compare for example NO2, not only with Josene measurements, but also with official RIVM values.

      Also the data is available through all SE OGC APIs, for example the SensorThings API.

      ad 6) The moment of truth! How well does the SE-based SE Stetl Python calibration results fit with the original RIVM values? One of the advantages of Data Harvesting (opposed to data push) is that we can switch back in time, i.e. restart harvesting from a given date. Harvesting and continuous calibration was restarted from august 1, 2018, the start of the calibration period at the RIVM station. Using a Grafana panel that displays both RIVM and SE-calculated values we can graphically see how well the data aligns.

      What we can see from the above image, is that visually the data aligns very well, here for NO2. The purple graph is the official RIVM measurement. Only station ASE NL 02 is not completely aligned.

      To also have some numeric proof and a more objective comparison, I dived in scatterplot and numerical analysis in Python. Apart from scatterplots that show calculated (Y) agains RIVM ref values (X) I calculated the “R-squared” and “slope” for fitting indicator values. This was also my first serious use of Python libs like Scipy, Pandas, Seaborn and Numpy (you’re never too old to become a data-scientist!).

      As all SE calibrated data is also stored in InfluxDB with RIVM refdata harvested from their SOS, it was easy to fetch values for the plots/calculations..

      Objectivity could be effected since station ASE NL 01 was finally deployed (okt 2018) in Nijmegen, also next to an RIVM station. So the calibration calculations from RIVM refdata in Breukelen could be compared to “Nijmegen”. The implementation for making these scatterplots can be found here. Lets look at some results, mainly for NO2, as I consider this one of the most important AQ indicator gasses.

      I like this image a lot as it shows an almost ideal alignment with an R2 of 0.976 and slope of almost 1. Mind: calibration was thus done at a very different site (about 80 km west) and AQ condition (highway) as the deployment (city street). 

      Above are plots for the other gasses as well. First row in Breukelen (no ref CO available in RIVM SOS), front row in Nijmegen. Only NO in Nijmegen is a bit problematic.

      To close off: this last image above shows NO2 fit at the Breukelen station for all five ASE boxes. Also quite good.

      What to conclude? First of all AirSensEUR is a major step forward in affordable accurate AQ sensing. We hope to expand the community.

      AlphaSense NO2 electrochemical sensors appear quite accurate, but calibration requires quite some effort, plus calibration formulas apply per individual sensor. Would automatic per-sensor ANN be less time-consuming and still accurate? Something I would like to investigate.

      The Smart Emission project and platform is still going strong, running within a Kubernetes Cloud maintained by Dutch Kadaster.

      Next emit will discuss how I integrated data from the amazing Luftdaten.info project for the municipality of Nijmegen.

      Categories: OSGeo Planet

      gvSIG Team: gvSIG 2.5 RC1 disponible, primera distribución candidata a final

      OSGeo Planet - Tue, 2019-02-19 13:13

      Ya está disponible para descargar la primera distribución candidata a final (RC1) de la nueva versión gvSIG 2.5.

      Esta versión puede descargarse desde el apartado de versiones en desarrollo de la web del proyecto.

      Las novedades que incluye esta versión son múltiples, desde nuevas funcionalidades hasta mejoras en rendimiento y estabilidad. Muchas de ellas pueden ser consultadas ya en los diferentes post que se están publicando en el blog, como pueden ser:

      Seguiremos publicando más posts sobre el resto de novedades durante las próximas semanas.

      Con la publicación de esta primera RC, pretendemos que la Comunidad gvSIG comience a probarla de forma masiva y que nos vayáis comunicando los errores que detectéis a través de las listas de usuarios para corregirlos.

      También estamos preparando en paralelo la versión para MAC OS X, cuya versión portable, así como las versiones portables de Windows y Linux, esperamos tenerlas en los próximos días.

      Categories: OSGeo Planet

      gvSIG Team: gvSIG 2.5 RC1 available, first release candidate to final version

      OSGeo Planet - Tue, 2019-02-19 13:12

      The first release candidate (RC1) of the new gvSIG 2.5 version is now available to download.

      This version can be downloaded from the development versions section of the project website.

      The novelties included in this version are multiple, from new functionalities to performance and stability improvements. Many of them can be consulted already in the different posts that are being published on the blog, such as:

      We will continue publishing more posts about the rest of the novelties during the next weeks.

      Publishing this first RC, we intend that the gvSIG Community start to test it in a massive way and that you send us the errors that you detect through the user mailing lists to be fixed.

      We are also preparing the version for MAC OS X in parallel, and its portable version, as well as the portable versions for Windows and Linux, will be available in the coming days.

      Categories: OSGeo Planet

      gvSIG Team: Towards the new gvSIG 2.5 version: Calling for translators

      OSGeo Planet - Mon, 2019-02-18 11:33

      The stabilization phase of the new gvSIG 2.5 version has started, and in order to have the final version with as many languages as possible the update of the translations of the interface are being carried out too.Currently gvSIG has been translated to more than 25 languages, thank to the participation of the Community.

      There are two ways of collaboration: updating any of the existing languages, as well as translating gvSIG interface to a new language in which gvSIG is not translated currently.

      If you are interested in translating gvSIG interface you can contact the gvSIG Project [info@gvsig.com] and we would send the instructions to do it.

      Translations that aren’t finished when the final version is released will be published as a package to be loaded from the gvSIG Add-ons manager. We will inform about these updates at the gvSIG blog.

      We expect your collaboration!

      Categories: OSGeo Planet

      gvSIG Team: Hacia la nueva versión gvSIG 2.5: Llamamiento a traductores

      OSGeo Planet - Mon, 2019-02-18 11:22

      La nueva versión 2.5 de gvSIG Desktop ya se encuentra en fase de estabilización, para lo cual se ha comenzado también con la actualización de las traducciones de la interfaz. El objetivo es el de tener la versión final disponible en el mayor número de idiomas posible.Actualmente gvSIG se encuentra traducido a más de 25 idiomas, gracias a la participación de la Comunidad.

      Existen dos formas de colaboración, bien actualizar cualquiera de los idiomas ya existentes en versiones anteriores, o bien traducir a algún idioma en el que no se encuentre traducido gvSIG actualmente.

      Si estáis interesad@s en traducir la interfaz de gvSIG podéis poneros en contacto con el proyecto [info@gvsig.com], y os indicaremos las instrucciones.

      Las traducciones que no estén finalizadas en el momento de la publicación de la versión final se publicarán como paquetes que podrán ser cargados directamente desde el Administrador de complementos de gvSIG. Iremos informando sobre estas actualizaciones en el blog de gvSIG.

      ¡Esperamos vuestra colaboración!

      Categories: OSGeo Planet

      gvSIG Team: Towards gvSIG 2.5: Report by point

      OSGeo Planet - Fri, 2019-02-15 11:25

      We are going to show you another novelty of the next version of gvSIG Desktop, that is called report by point. It is a configurable multilayer information button, and it is very useful to consult the information of different layers on a concrete location and visualize it in a simultaneous way.

      The operation is very simple, you only have to click on a specific point on the View and a report will be shown in a new window on that point with all the attributes of all visible layers. In that report you can select and copy data to the clipboard, being able to paste them in other applications.

      In addition, a new tab has been added to the layer “Properties”, that allows to define which fields you want to see in the report, and even you can modify the field name to be shown.

      In the following video you can see how it works. Simple and extremely useful.

      Categories: OSGeo Planet

      gvSIG Team: Camino a gvSIG 2.5: Informe por punto

      OSGeo Planet - Fri, 2019-02-15 10:01

      La novedad que os traemos hoy de la próxima versión de gvSIG Desktop la hemos denominado informe por punto. Viene a ser un botón de información multicapa y configurable. ¡Muy útil para consultar la información en un punto de distintas capas y visualizarla de forma simultanea!

      El funcionamiento es muy sencillo, pulsamos en un punto determinado de la Vista y se nos mostrará en una nueva ventana un informe sobre ese punto con todos los atributos de todas las capas visibles. En este informe podemos seleccionar y copiar los datos al portapapeles, pudiendo así luego pegarlos en otras aplicaciones.

      Además se ha añadido una nueva pestaña a las “Propiedades” de la capa que permite definir que campos queremos que se visualicen en el informe, e incluso modificar el nombre con que aparece el campo.

      En el siguiente vídeo podéis ver su funcionamiento. Sencillo y extremadamente útil.

      Categories: OSGeo Planet

      gvSIG Team: Free course about Geographic Information Systems applied to archaeology: Certification and links to the complete course

      OSGeo Planet - Thu, 2019-02-14 14:19

      The certification of the course about Geographic Information Systems applied to archaeology is now available (this course is also available in Spanish).

      This certification starts after publishing the seven modules that form the course, but it will continue open continuously, so that any user can get it at any moment when the modules are finished.

      To take the course, that is free of charge, it’s not necessary any registration. You only have to follow the videos of the different modules induicated at the end of this post.

      In addition, if you want to get the certification of the course, that is optional, a complete exercise must be completed, which includes some of the contents shown during the course. Likewise, you must have at least 6 of the 9 existing activities correctly, which will validate the knowledge acquired during the course and will be evaluated by a tutor.

      Apart from the delivery and passing exercise successfully, the certification will have a minimum cost, necessary to cover the expenses related to the evaluation and certification. This cost will be € 25.

      The certification will be issued by the gvSIG Association, and will include the Course completion certificate.

      The exercise and the cartography to be used at this practical exercise can be downloaded from the following links:

      In the first section of the practical exercise the steps to follow to send it are explained in detail, as well as how to make the payment.

      The course is composed of the following modules:

      Categories: OSGeo Planet

      gvSIG Team: Curso gratuito de Sistemas de Información Geográfica aplicados a arqueología: Certificación y enlaces al curso completo

      OSGeo Planet - Thu, 2019-02-14 14:11

      Ya está disponible la certificación del curso de Sistemas de Información Geográfica aplicados a arqueología (este curso también está disponible en inglés).

      Esta certificación se abre tras la publicación de los siete módulos que componen el curso, pero seguirá abierta de forma continua, por lo que cualquier usuario podrá obtenerla en el momento en que finalice los distintos módulos.

      Para poder realizar el curso, que es gratuito, no es necesario registrarse en ningún portal, solo se deben seguir los vídeos de los módulos que lo forman, indicados al final de este post.

      Por otro lado, si se desea obtener la certificación del curso, que es opcional, se deberá completar un completo ejercicio, que incluye algunos de los contenidos impartidos durante el curso. Así mismo, se deberá tener correctamente al menos 6 de las 9 actividades existentes en dicho ejercicio, que validará los conocimientos adquiridos durante el curso y será evaluado por un tutor.

      Aparte de la entrega y aprobación del ejercicio, la certificación llevará un coste mínimo, necesario para cubrir los gastos relativos a la evaluación y certificación. Este coste será de 25.

      La certificación será emitida por la Asociación gvSIG, e incluirá el Certificado de aprovechamiento del curso.

      El ejercicio y la cartografía a utilizar en el mismo pueden descargarse desde los siguientes enlaces:

      En el primer apartado del ejercicio práctico se explica con detalle los pasos a seguir para enviar la práctica, así como para la realización del pago para poder obtener el certificado.

      El curso se compone de los siguientes módulos:

      Categories: OSGeo Planet

      gvSIG Team: Towards gvSIG 2.5: “Indigenous mapping” symbol library

      OSGeo Planet - Wed, 2019-02-13 13:24

      Not all the novelties of the next gvSIG Desktop version are new functionalities. We added a new symbol library called “Indigenous mapping” that contains an interesting set of point symbols developed by “The Ethnographic Mapping Lab” of the University of Victoria and designed by the graphic artist James Gray.

      The objective of these icons is to represent the set of activities, resources and locations commonly associated with indigenous mapping projects. We hope that they are useful for you.

      Its installation and use is very simple, as you can see in the following video:

      Categories: OSGeo Planet

      gvSIG Team: Camino a gvSIG 2.5: Biblioteca de símbolos “Indigenous mapping”

      OSGeo Planet - Wed, 2019-02-13 13:24

      No todas las novedades de la próxima versión de gvSIG Desktop son nuevas funcionalidades. Añadimos una nueva biblioteca de símbolos denominada “ Indigenous mapping” que contiene un interesante conjunto de símbolos puntuales desarrollados por “The Ethnographic Mapping Lab” de la Universidad de Victoria y diseñados por el artista gráfico James Gray.

      El objetivo de estos iconos es representar el conjunto de actividades, recursos y localizaciones comúnmente asociadas a los proyectos de mapeo indígena. Esperamos que os resulten de utilidad.

      Su instalación y uso es muy sencilla, como podréis comprobar en el siguiente vídeo:

      Categories: OSGeo Planet

      Fernando Quadro: GeoCast Live: OpenStreetMap no apoio ao desastre de Brumadinho

      OSGeo Planet - Wed, 2019-02-13 11:28

      No dia 14/02 as 19h acontecerá uma live promovida pelo Canal GeoCast (YouTube). Nesta live será possível entender como se deu o esforço colaborativo de mapeamento da comunidade OpenStreetMap (OSM) na montagem de um mapa base pré e pós desastre da barragem de rejeitos de minério de ferro, da mineradora Vale em Brumadinho-MG, com o intuito de auxiliar no planejamento das buscas e do resgate das vítimas, bem como na recuperação a longo prazo da cidade de Brumadinho.

      Se você tem interesse de participar, tirar dúvidas, ou simplesmente entender como se deu esse esforço colaborativo, segue abaixo o link para a live:

      Categories: OSGeo Planet

      gvSIG Team: Camino a gvSIG 2.5: Guardar y cargar un etiquetado

      OSGeo Planet - Wed, 2019-02-13 07:28

      Seguimos con más novedades. En este caso os traemos una pequeña funcionalidad pero muy útil. Se trata de la posibilidad de guardar una configuración de etiquetado y poder recuperarla para aplicarla posteriormente.

      Cualquier usuario de un SIG sabe que en ocasiones puede conllevar un tiempo considerable definir las características de un etiquetado; una configuración de etiquetado que no es recuperable para otros proyectos. Y también puede ocurrir que en un mismo proyecto, a una misma capa, nos pueda interesar aplicarle distintos tipos de etiquetado.

      Este tipo de necesidades se cubren con esta nueva funcionalidad que permite guardar la configuración del etiquetado y recuperarla siempre que lo necesitemos.

      Os dejamos con una pequeño demo de su funcionamiento:

      Categories: OSGeo Planet
      Syndicate content