ENI (2) ENI Document, is it operational? Verifying documents and "expedients". Afirma and TipoFirmasElectronicas

0. Introduction
1. Our XML validator
2. Main problems
3. Adapting Afirma to TipoFirmasElectyronicas
4. Oficial validators

0. Introduction

Spanish Government has made some small steps to solve the situation. But it is not enough. 
For instance, there are no available schema validators for signatures provided by the program "AutoFirma".

I have spent a lot of time trying to create ENI "expedients" and documents. 

1. Our own XML validator 

You should avoid using Jackson XML. Use JAXB instead. This small program validates the XML to the schemas
========================================
package openadmin.utils.jaxb;

import javax.xml.XMLConstants;
import javax.xml.transform.stream.StreamSource;
import javax.xml.validation.Schema;
import javax.xml.validation.SchemaFactory;
import javax.xml.validation.Validator;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.StringReader;
import java.io.StringWriter;


import org.apache.commons.io.IOUtils;
import org.xml.sax.SAXException;

import jakarta.xml.bind.JAXBContext;
import jakarta.xml.bind.JAXBElement;
import jakarta.xml.bind.JAXBException;
import jakarta.xml.bind.Marshaller;
import jakarta.xml.bind.Unmarshaller;
import openadmin.utils.jackson.IJacksonEdu;


public class JAXBEduImp implements IJacksonEdu{
	
	public JAXBEduImp() {
		
	}
	
	public Marshaller getMarshaller(Class myClass) throws JAXBException {
		JAXBContext context = JAXBContext.newInstance(myClass);
        Marshaller m = context.createMarshaller();
        m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
        return m;
	}
	
	public Unmarshaller getUnmarshaller(Class myClass) throws JAXBException {
		JAXBContext context = JAXBContext.newInstance(myClass);
		Unmarshaller jaxbUnmarshaller = context.createUnmarshaller();  
        return jaxbUnmarshaller;
	}
	
	public String ObjectToString (Object obj) throws Exception {
		Marshaller m = getMarshaller(obj.getClass());
		StringWriter sw = new StringWriter();
	    m.marshal(obj, sw);
	    String result = sw.toString();
        return result;
	}
	
	
	public void ObjectToFile (Object obj, String fileName) throws Exception {
		Marshaller m = getMarshaller(obj.getClass());
		m.marshal(obj, new FileOutputStream(fileName));
	}
	
	public StringWriter ObjectToStringWriter (Object obj) throws Exception {
		Marshaller m = getMarshaller(obj.getClass());
		StringWriter sw = new StringWriter();
		m.marshal(obj, sw);
		return sw;
	}
	
	public InputStream ObjectToIputStream(Object obj) throws Exception {
		return IOUtils.toInputStream(this.ObjectToString(obj), "UTF-8");
	}
	
	public <T extends Object> T InputStreamToObject(InputStream in, Class<T> valueType) throws Exception {
		Unmarshaller unM = getUnmarshaller(valueType);
		T obj = (T) unM.unmarshal(in);
		return obj;
		//JAXBElement<T> root = (JAXBElement<T>) unM.unmarshal(in);
		//return root.getValue();
	}
	
	public <T extends Object> T StringToObject (String content, Class<T> valueType) throws Exception  {
		Unmarshaller unM = getUnmarshaller(valueType);
		StringReader reader = new StringReader(content);
		T obj = (T) unM.unmarshal(reader);
		return obj;
		//JAXBElement<T> root = (JAXBElement<T>) unM.unmarshal(reader);
		//return root.getValue();
	}
	
	public <T extends Object> T ByteArrayToObject (byte[] content, Class<T> valueType) throws Exception{
		Unmarshaller unM = getUnmarshaller(valueType);
		InputStream inStrm = new ByteArrayInputStream(content);
		T obj = (T) unM.unmarshal(inStrm);
		return obj;
		//JAXBElement<T> root = (JAXBElement<T>) unM.unmarshal(inStrm);
		//return root.getValue();
	}
	
	public <T extends Object> T FileToObject (File src, Class<T> valueType) throws Exception  {
		Unmarshaller unM = getUnmarshaller(valueType);
		//T obj= (T) unM.unmarshal(src);  //Peta!!
		//return obj;
		JAXBElement<T> root = (JAXBElement<T>) unM.unmarshal(src);
		return root.getValue();
	}
	
	public <T extends Object> T FileToObject (InputStream src, Class<T> valueType) throws Exception {
		Unmarshaller unM = getUnmarshaller(valueType);
		T obj = (T) unM.unmarshal(src);
		return obj;
		//T obj= (T) unM.unmarshal(src);  
		//return obj;
	}
	
	
	
	/**
	 * 
	 * @param xmlFile
	 * @param schemaFile is relative from resource folder
	 * @return
	 */
	public boolean validateXML(String xmlFile, String schemaFile) {
        SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
        Schema schema = null;
        try {
            try {
            	schema = schemaFactory.newSchema(
            		new File(Thread.currentThread().getContextClassLoader().getResource(schemaFile).getPath()));
            } catch (Exception e1) {
            	schema = schemaFactory.newSchema(new File(schemaFile));
            }
            Validator validator = schema.newValidator();
            validator.validate(new StreamSource(new File(xmlFile)));
            return true;
            
        } catch (SAXException | IOException e) {
            e.printStackTrace();
            return false;
        }
    }
	
	public static void main (String[] args) {
		
		boolean ok= new JAXBEduImp().validateXML("/home/eduard/prova_decrets/V1/2021/0ExpedientENI.xml",
				"/home/eduard/WORKSPACES/WS_GRDL01/Ajuntament/A10-xsd2Pojo/src/main/resources/xsd/expedienteEni.xsd");
		/*
		boolean ok= new JAXBEduImp().validateXML("/home/eduard/prova_decrets/V1/2021/ES_L01462384_2021_DOC_RESOLUCIO_0005_ENI.xml",
				"/home/eduard/WORKSPACES/WS_GRDL01/Ajuntament/A10-xsd2Pojo/src/main/resources/xsd/documentoEni.xsd");
		
		/*
		boolean ok1= new JAXBEduImp().validateXML("/home/eduard/Baixades/ES_L01462384_2022_DOC_12473165_0000_Pressupostampl.xml",
				"/home/eduard/WORKSPACES/WS_GRDL01/Ajuntament/A10-xsd2Pojo/src/main/resources/xsd/documentoEni.xsd");
		*/
		
		if (ok) System.out.println("OK");
		else System.out.println("ERROR!");
	}
	
	
}
========================================

2. Main problems

This validator (and not the referenced in the obsolete parts of this blog) can detect some problems like duplicity of "ids", and malformed "ids" and "references".

The malformed "ids" and "references" are produced by using spaces and some characters like accents and other symbols.

You should also change some field types as mentioned in the previous post.

3. Adapting the class AFirma to TipoFirmasElectronicas

Here is a function to adapt these 2 classes
 
====================================================

public static List<TipoFirmasElectronicas> getENIFirmasCertificado (AFIRMA aFirma, String idContenido, int any) {
    //Firmas firmas=new Firmas();
    int i=1;
    List<TipoFirmasElectronicas> lFirmasElectronicas= new ArrayList<TipoFirmasElectronicas>();
		
    //for (SignatureType signature: aFirma.getFirmas()) {
    for (SignatureType signature: aFirma.getSignature()) {
	TipoFirmasElectronicas firmaElectronica= new TipoFirmasElectronicas();
	firmaElectronica.setId("firma:"+i);
	firmaElectronica.setRef("#"+ idContenido);
	firmaElectronica.setTipoFirma("TF02"); // De moment gexflow firma en Xades internally detached pero no ho tinc clar
	firmaElectronica.setId("Firma_"+ any+"_"+String.format("%04d", i));
			
	ContenidoFirma contenidoFirma=new ContenidoFirma();
				
	// FimaConCertificado a elegir entre (byte[] firmaBase64, SygnatureType signature (que es pot repecar de la classe aFirma), Object referenciaFirma 
	FirmaConCertificado firmaConCertificado = new FirmaConCertificado();
	firmaConCertificado.setSignature(signature);
	//No gastem firmaConCertificado.setFirmaBase64();
	//No gastem firmaConCertificado.setReferenciaFirma();
				
	contenidoFirma.setFirmaConCertificado(firmaConCertificado);
	firmaElectronica.setContenidoFirma(contenidoFirma);
	lFirmasElectronicas.add(firmaElectronica);
	i++;
    }
    return lFirmasElectronicas;

 }

====================================================

Note that TipoDocumento.getFirmas().getFirma() is of type List<TipoFirmasElectronicas>

4. Official ENI validators

Official validators only verify if the file is using the "xsd "schema or not. But don't inform about the type of the error or its position. So it is important that you use a good schema validator as the JAXB (not the JaksonXML) as the one in the previous post.

4.1 INSIDE validator

The INSIDE validator can validate "expedients" and documents. For "expedients"


It also verifies the signature and "organic entity codes".

The "organic entity codes" can be rather easily defined as usually, the same "entity code" is used.

But the signature is a bit more problematic. There are some signature formats that are official (like CADES explicit) that are not allowed in "expedients".

See this post to use AutoFirma for signing with CADES implicit

To validate documents



but this time no information about signature validation is shown!

So INSIDE may not be a good option for validations documents!

4.2 Sede electrónica de administracion.gob.es validator

This validator can validate both documents and "expedients". But the messages are not very expressive!

For "expedients" it can validate also the "codigo procedimiento SIA"


For documents:

Spanish Government is not acting well. After more than 13 years, a good ENI validator is missing. A good ENI validator must have these facilities:

1. Tell if the file is good or not
2. Verify XML schemas, signatures, SIA procedures, entities, etc.
3. Show where is the error in the document and what type of error has been produced
4. Show information on how to solve the error 

I think this "simple" project is very important and it is not very expensive to develop!

Spain and Europe governments are being manipulated by lobbies and investor groups and the citizens are paying a great cost with these types of politics. 




0. Introduction (OBSOLETE!!!) use JAXB validator instead!!!

Let's try to use the modified schema in the previous post, and try to validate some simple ENI document examples that politely offers La Junta de Andalucia

I have downloaded them and used this class to validate them:
 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
package utils;

import javax.xml.XMLConstants;
import javax.xml.transform.stream.StreamSource;
import javax.xml.validation.Schema;
import javax.xml.validation.SchemaFactory;
import javax.xml.validation.Validator;
import java.io.File;

import java.io.IOException;
import org.xml.sax.SAXException;


import com.fasterxml.jackson.dataformat.xml.XmlMapper;
import com.fasterxml.jackson.module.jaxb.JaxbAnnotationModule;

public class XMLUtils {
 
 /**
  * Return a Jackson XML Mapper
  * @param defaultUserWrapper for accepting collections wrappers (For ORVE set to false)
  * @param acceptJaxbAnnotations for accepting Jaxb annotations (Fort ORVE set to true) 
  * @return
  */
 public static XmlMapper getXmlMapper(boolean defaultUserWrapper, boolean acceptJaxbAnnotations) {
  XmlMapper mapper = new XmlMapper();
  mapper.setDefaultUseWrapper(defaultUserWrapper); 
  if (acceptJaxbAnnotations) mapper.registerModule(new JaxbAnnotationModule()); // Accept JAXB Annotations !!!!
  return mapper;
 }
 
 /**
  * Return a Jackson XML Mapper suitable for ORVE that:
  *  doesn't accept collections wrappers
  *  accepts Jaxb annotations
  * @return
  */
 public static XmlMapper getXmlMapper() {
  return getXmlMapper(false, true);
 }
 
 /**
  * 
  * @param xmlFile
  * @param schemaFile is relative from resource folder
  * @return
  */
 public static boolean validateXML(String xmlFile, String schemaFile) {
        SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
        try {
            Schema schema = schemaFactory.newSchema(new File(FileUtilsEdu.getPathFromResourcesFolder(schemaFile)));

            Validator validator = schema.newValidator();
            //validator.validate(new StreamSource(new File(getResource(xmlFile))));
            validator.validate(new StreamSource(new File(xmlFile)));
            return true;
            
        } catch (SAXException | IOException e) {
            e.printStackTrace();
            return false;
        }
    }
 
 
 
 public static void main(String[] args) {
  
 
  //boolean isValid=validateXML("/home/eduard/Angela/2018ContractesMenors.xml","xsd/contractes_menors.xsd");
  boolean isValid=validateXML("/home/ximodante/ENI01.xml","xsd/documentoEni.xsd");
  System.out.println("ENI01.xml-->"+ isValid);
  
  isValid=validateXML("/home/ximodante/ENI02.xml","xsd/documentoEni.xsd");
  System.out.println("ENI02.xml-->"+ isValid);
  
 
 }
 
}


But the process of validation needs more than 30 seconds per file in the local machine!!!

Do you think this system is operational?


PS: Here is the code of another class you need



  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
package utils;

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
//import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
//import java.util.regex.Pattern;

//import javax.faces.context.FacesContext;

import org.apache.commons.lang3.StringUtils;

public class FileUtilsEdu {

 /**
  * Read delimited data from a delimited file  
  * @param pFileName the name of the file
  * @param pDelimiter  The delimiter of the data
  * @return 
  * @throws FileNotFoundException
  * @throws IOException
  */
 public static List<String[]> ReadData(String pFileName, String pDelimiter, String commentStr) throws FileNotFoundException, IOException{
  
  BufferedReader inputStream = Files.newBufferedReader(Paths.get(pFileName));
  
  //Pattern p = Pattern.compile(pDelimiter);
  List<String[]> lst=new ArrayList<String[]>();
  while(true) {
   String strLine= inputStream.readLine();     // Input a line from the file
   if (strLine == null) break;        // We have reached eof
   System.out.println(strLine);
   strLine=strLine.trim();
   strLine=StringUtils.substringBefore(strLine,commentStr);
   //if (strLine.length()>1) lst.add(p.split(strLine)); // Split the line into strings delimited by delimiters
   //if (strLine.length()>1) lst.add(strLine.split(pDelimiter)); // Split the line into strings delimited by delimiters
   if (strLine.length()>1) lst.add(StringUtilsEdu.splitAndTrim(strLine,pDelimiter)); // Split the line into strings delimited by delimiters
  }
  
  return lst;
 } 
 
 
 /**
  * Get an input stream from a relative path from WebContent folder in NO Maven Project
  * 
  * If there is a file in the folder WebContent/resources/data/AccessData.txt we should call it as:
  * 
  *   Inputstream in=getStreamFromWebContentFolder("./resources/data/AccessData.text")
  * 
  * @param filePath
  * @return
  */
 
 //public static InputStream getStreamFromWebContentFolder(String filePath) {
 // return FacesContext.getCurrentInstance().getExternalContext().getResourceAsStream(filePath);
 //}
 
 /**
  * Get the path from a relative path from WebContent folder in NO Maven Project
  * 
  * If there is a file in the folder WebContent/resources/data/AccessData.txt we should call it as:
  * 
  *   InputStream in=getPathFromWebContentFolder("./resources/data/AccessData.text")
  * 
  * @param filePath
  * @return
  * @throws MalformedURLException 
  */
 //public static String getPathFromWebContentFolder(String filePath) throws MalformedURLException {
 // return FacesContext.getCurrentInstance().getExternalContext().getResource(filePath).getPath();
 //}
 
 /**
  * Get an input stream from a relative path from resources folder in Maven Project
  * 
  * If there is a file in the folder resources/data/AccessData.txt we should call it as:
  * 
  *   InputStream in=getStreamFromResourcesFolder("data/AccessData.text")
  * 
  * @param filePath
  * @return
  */
 public static InputStream getStreamFromResourcesFolder(String filePath) {
  return Thread.currentThread().getContextClassLoader().getResourceAsStream(filePath);
 }
 
 
 /**
  * Get the path from a relative path from resources folder in Maven Project
  * 
  * If there is a file in the folder resources/data/AccessData.txt we should call it as:
  * 
  *   String path=getPathFromWebContentFolder("data/AccessData.text")
  * 
  * @param filePath
  * @return
  * @throws MalformedURLException 
  */
 public static String getPathFromResourcesFolder(String filePath) {
  return Thread.currentThread().getContextClassLoader().getResource(filePath).getPath();
 }
 
 //Thread.currentThread().getContextClassLoader().getResource("").getPath();
 
 public static void main(String[] args) {
  // TODO Auto-generated method stub

 }

}

1. FALSE Validation of ENI documents (OBSOLETE!!)


I have found an official ENI document validator that works on old java version 1.5. I have validated some ENI good examples and find no errors with this validator. But I am a bit surprised as it is relatively quick.

Having a look at the sources I cannot see any references to "signing libraries", so it made me suspect that NO SIGNATURE validation is done. Hence I have built an ENI document whose signature part is borrowed from another document, and NO PROBLEMS have been detected.
So this is only a SYNTACTIC validation.

There is an official ENI validator online but does not work. It states that there are continuous problems:

Again, desperation arises due to the waste of valuable time!

Do you think the Spanish Government is acting correctly with this stuff?

I don't think so! Only good appearance but no reliable solutions!

The ENI specification is from year 2010. There has been 9 years from now but we have no satisfying solution for validating ENI documents







Comments

Post a Comment

Popular posts from this blog

ORVE WS (Dynamic) (4) Jackson XML mapper

0. Introduction I love Jackson parser as it is easy to use and can be used to with some data formats such as CSV ,  JSON , YAML and XML . Just define a wrapper and a class for marshalling/unmarshalling. Although JAXB is a standard, in the Java ecosystem exists many solutions. And my decision is to use Jackson. 1. Defining MAVEN dependencies To use Jackson XML it is required this dependency in Maven. Note that  ${jackson.version} represents a variable that contains preferably the latest Jackson version. 1 2 3 4 5 <dependency> <groupId> com.fasterxml.jackson.dataformat </groupId> <artifactId> jackson-dataformat-xml </artifactId> <version> ${jackson.version} </version> </dependency> But if you want to use CSV, JSON or YAML in the project, it is as easy as including these dependencies 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 <!-- Jackson for CSV --> <de...
Read more

ENI (1) ENI Document OR the Spanish Electronic Administration Mafia

Image
0.Introduction. Getting the "xsd" schemas 1. Obtaining the afirma.xsd 2. Managing the "xsd" schemas. Changing  schemalocation  property from the  xsd  tag 3. Generating classes from xsd schema 4. Modifying the classes to solve operational problems 0. Old version 1. Generating the classes 0.Introduction. Getting the "xsd" schemas To obtain the correct URLs of the schemas: 1.For the document: https://administracionelectronica.gob.es/ENI/XSD/v1.0/documento-e in this URL it is shown these schemas (click on the link and download the schemas): http://administracionelectronica.gob.es/ENI/XSD/v1.0/documento-e/documentoEni.xsd http://administracionelectronica.gob.es/ENI/XSD/v1.0/documento-e/metadatos/metadatosDocumentoEni.xsd http://administracionelectronica.gob.es/ENI/XSD/v1.0/documento-e/contenido/contenidoDocumentoEni.xsd http://administracionelectronica.gob.es/ENI/XSD/v1.0/firma/firmasEni.xsd 2. For the file (expediente) https://administracionelectronica.gob.es...
Read more

ORVE WS (Dynamic) (12) Defiining the control tables

0. Introduction Our goals are: Retrieving the INPUT records (Received from other administrations) Retrieving the OUTPUT records (Sent to other administrations) To retrieve any record the WS requires at least these parameters   1. State (Estado). For INPUT these are the available states: RPC : Received but confirmation pending RR : Received and rejected RC : Received and confirmed RREEN : Received and resent      For OUTPUT : PE : Pending delivery EPC : Sent but confirmation is pending ER : Sent but rejected EC : Sent and confirmed 2. Date from in " YYYY-MM-DD HH:MM:SS " format 3. Date to in the same format. The system knows who is our administration (Oficina or Office) 1. Additional tables or entities We need to have control of: "Ids" that can identify the records and if the records have been saved and passed to the Gexflow application (an application that controls our records). To get the "Ids", the WS operation...
Read more