Topic: QElectrotech is freezing / loading time library elements

QElectroTech isn't alone in using file / element strategy
-KiCAD
    footprints +10500 file's in 151 folder's
    packages3D +11400 file's in 100 folder's
    symbols 490 file's in 23 folder's => multiple element per file maybe there is performance problem after all
-eagle
    symbols => multiple element per file maybe there is performance problem after all
   
I will check with the neighbors (KiCAD) if they are also bothered by this and how this has been remedied.

Has SQLite been considered for the element?

Or maybe we should also put multiple elements in 1 file.

  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

./count_elements.sh
7321 elements dans 1023 categories (soit 8344 fichiers)

Post's attachments

Attachment icon count_elements.sh 246 b, 278 downloads since 2020-07-24 

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."
  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

https://git.tuxfamily.org/qet/qet.git/c … 2289c9d393
https://git.tuxfamily.org/qet/qet.git/c … e0d3bb1186

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."

Re: QElectrotech is freezing / loading time library elements

Could this be a cause of concern, performance wise?
@qet/sources/ElementsCollection/elementcollectionhandler.cpp

ElementsLocation ECHSFileToFile::copyElement(ElementsLocation &source, ElementsLocation &destination, const QString& rename)
{
    QString new_elmt_name = rename.isEmpty() ? source.fileName() : rename;
    bool rb = QFile::copy(source.fileSystemPath(), destination.fileSystemPath() + "/" + new_elmt_name);
    if (rb)
    {
#ifdef Q_OS_WIN
        //On windows when user drag and drop an element from the common elements collection
        //to the custom elements collection, the element file stay in read only mode, and so
        //user can't save the element
        extern Q_CORE_EXPORT int qt_ntfs_permission_lookup;
        qt_ntfs_permission_lookup++;
        QFile file(destination.fileSystemPath() + "/" + new_elmt_name);
        if (!file.isWritable()) {
            if (!file.setPermissions(file.permissions() | QFileDevice::WriteUser)) {
                qDebug() << "Failed to change file permission of : " << QFileInfo(file).canonicalFilePath() \
                         << " in ECHSFileToFile::copyElement";
            }
        }
        qt_ntfs_permission_lookup--;
#endif
        return ElementsLocation (destination.fileSystemPath() + "/" + new_elmt_name);
    }
    else
        return ElementsLocation();
}

5 Reply by Joshua (edited by Joshua 2020-07-24 18:51:38)

  • Joshua
  • Joshua
  • QElectroTech Team Developer
  • Offline

Re: QElectrotech is freezing / loading time library elements

No, this part of code is used when you drag an element from the common collection to the user collection and so is not called when the collection is loaded or when user drag and drop element into the folio.

Développeur QElectroTech
  • Joshua
  • Joshua
  • QElectroTech Team Developer
  • Offline

Re: QElectrotech is freezing / loading time library elements

De-Backer wrote:

QElectroTech isn't alone in using file / element strategy
-KiCAD
    footprints +10500 file's in 151 folder's
    packages3D +11400 file's in 100 folder's
    symbols 490 file's in 23 folder's => multiple element per file maybe there is performance problem after all
-eagle
    symbols => multiple element per file maybe there is performance problem after all
   
I will check with the neighbors (KiCAD) if they are also bothered by this and how this has been remedied.

Has SQLite been considered for the element?

Or maybe we should also put multiple elements in 1 file.

May be one file for multiple element should be a good way, but the code of qet isn't write for that and write something to handle this will take a lot lot of time and make code very un-understandable, so no.

We already use a sqlite db for element icon in the widget of element collection, but the default is that Qt sqlite isn't multithreade and in definitive load is faster without the db even in linux.

I'm playing with a win10 vm, for the moment I find nothing to improve file access.....
I you have some links about how Kicad or other softwares load multiple files I will be glade to read it.
Not easy this problem.

Développeur QElectroTech
  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

Maybe use Msvc compiler, if someone can try it?

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."
  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

https://forum.qt.io/topic/109709/speed- … ication/17

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."

Re: QElectrotech is freezing / loading time library elements

Joshua wrote:

Not easy this problem.

Well that was to be expected that it would not be easy, nevertheless it is something that needs to be addressed. Now adding new features will only make the problem worse.

So I only have 2 options to improve the Wiki or the code.
The program is used every day.
Anyway I give it a try, come I up empty, no problem.

10 Reply by De-Backer (edited by De-Backer 2020-07-24 20:25:28)

Re: QElectrotech is freezing / loading time library elements

the developer of KiCad likes to use "try & catch"

try
        {
            cache_lib = AddLibrary( cache_name );
 
            if( cache_lib )
                cache_lib->SetCache();
        }
        catch( const IO_ERROR& ioe )
        {
            wxString msg = wxString::Format( _(
                    "Symbol library \"%s\" failed to load.\nError: %s" ),
                    GetChars( cache_name ),
                    GetChars( ioe.What() )
                    );
 
            THROW_IO_ERROR( msg );
        }

https://gitlab.com/kicad/code/kicad/-/b … y.cpp#L550

not really and QT thing.

and  wxwidgets != QT
https://www.wxwidgets.org/

https://bugreports.qt.io/browse/QTBUG-1 … 22qfile%22

11 Reply by De-Backer (edited by De-Backer 2020-07-24 22:21:01)

Re: QElectrotech is freezing / loading time library elements

Has the QXmlStreamReader Class been considered?

https://doc.qt.io/qt-5/qxmlstreamreader.html

Multithreaded File I/O
https://www.drdobbs.com/parallel/multit … /220300055

for ref
https://programmer.group/efficiency-com … idxml.html

  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

We use pugixml https://pugixml.org/
https://qelectrotech.org/forum/viewtopi … 109#p11109

Some benchmarks :

Under my Debian Sid  devel machine :

Qtxml enable
Le chargement de la collection d'éléments à été éffectué en 798 ms

pugixml enable
Le chargement de la collection d'éléments à été éffectué en 471 ms


AppImage

Qtxml enable
e chargement de la collection d'éléments à été éffectué en 782 ms


pugixml enable
Le chargement de la collection d'éléments à été éffectué en 540 ms

https://github.com/zeux/pugixml/issues/96

https://git.tuxfamily.org/qet/qet.git/t … ig.hpp#n20

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."

Re: QElectrotech is freezing / loading time library elements

and what if we index when we open the folder.
instead of everything at start up?

  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

Humm try to compile with visual studio 2019 under Win10 ..
https://github.com/drawpile/Drawpile/wi … om-sources

[ 18%] Building CXX object src/lib/CMakeFiles/KF5CoreAddons.dir/text/kmacroexpander_win.cpp.obj
kmacroexpander_win.cpp
[ 19%] Building CXX object src/lib/CMakeFiles/KF5CoreAddons.dir/util/klistopenfilesjob_win.cpp.obj
klistopenfilesjob_win.cpp
[ 19%] Building CXX object src/lib/CMakeFiles/KF5CoreAddons.dir/util/kprocesslist_win.cpp.obj
kprocesslist_win.cpp
[ 20%] Building CXX object src/lib/CMakeFiles/KF5CoreAddons.dir/util/kshell_win.cpp.obj
kshell_win.cpp
[ 20%] Building CXX object src/lib/CMakeFiles/KF5CoreAddons.dir/util/kuser_win.cpp.obj
kuser_win.cpp
[ 21%] Building CXX object src/lib/CMakeFiles/KF5CoreAddons.dir/ECMQmLoader-kcoreaddons5_qt.cpp.obj
ECMQmLoader-kcoreaddons5_qt.cpp
[ 21%] Building CXX object src/lib/CMakeFiles/KF5CoreAddons.dir/kcoreaddons_debug.cpp.obj
kcoreaddons_debug.cpp
[ 22%] Linking CXX shared library ..\..\bin\KF5CoreAddons.dll
LINK Pass 1: command "C:\PROGRA~2\MICROS~1\2019\COMMUN~1\VC\Tools\MSVC\1426~1.288\bin\Hostx86\x86\link.exe /nologo @CMakeFiles\KF5CoreAddons.dir\objects1.rsp /out:..\..\bin\KF5CoreAddons.dll /implib:..\..\lib\KF5CoreAddons.lib /pdb:C:\Users\laurent\Downloads\kcoreaddons-master\kcoreaddons-master\build\bin\KF5CoreAddons.pdb /dll /version:5.73 /machine:X86 /debug /INCREMENTAL C:\Qt\5.15.0\msvc2019_64\lib\Qt5Cored.lib netapi32.lib userenv.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:CMakeFiles\KF5CoreAddons.dir/intermediate.manifest CMakeFiles\KF5CoreAddons.dir/manifest.res" failed (exit code 1112) with the following output:
Qt5Cored.lib(Qt5Cored.dll) : fatal error LNK1112: type d'ordinateur module 'x64' en conflit avec le type d'ordinateur cible 'x86'
NMAKE : fatal error U1077: '"C:\Program Files\CMake\bin\cmake.exe"' : code retour '0xffffffff'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x86\nmake.exe"' : code retour '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x86\nmake.exe"' : code retour '0x2'
Stop.

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."
  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

Ok, fixed..
nmake /I
nmake  install /I

Post's attachments

Attachment icon msvs-kf5correaddons.png 458.95 kb, 110 downloads since 2020-07-25 

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."

Re: QElectrotech is freezing / loading time library elements

scorpio810 wrote:

Humm try to compile with visual studio 2019 under Win10 ..
https://github.com/drawpile/Drawpile/wi … om-sources

Now wait what is the problem the code, the compiler or the OS

  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

https://forum.qt.io/post/520338
https://community.kde.org/Guidelines_an … ce/Windows
https://community.kde.org/KDEConnect/Build_Windows

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."
  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

Very complicated with Visual Studio and I prefer to stop, we'll see later.

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."

Re: QElectrotech is freezing / loading time library elements

scorpio810 wrote:

Very complicated with Visual Studio and I prefer to stop, we'll see later.

What were you trying to achieve?
as you can see:

De-Backer wrote:

Now wait what is the problem the code, the compiler or the OS

I don't understand the link.

  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

I tried to saw if QET run faster under Windows if we compile it with Qt for MSVC.

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."

Re: QElectrotech is freezing / loading time library elements

scorpio810 wrote:

I tried to saw if QET run faster under Windows if we compile it with Qt for MSVC.

ok
And what do you think of it, open of elements files

De-Backer wrote:

and what if we index when we open the folder.
instead of everything at start up?

eg if you open the folder "10_electric" only then index.

10_allpole  11_singlepole  20_manufacturers_articles  90_american_standards  91_en_60617  98_graphics  99_miscellaneous&unsorted  qet_directory  qet_labels.xml

=> all folders
open the folder 10_allpole

100_folio_referencing  114_connections    130_terminals&terminal_strips  200_fuses&protective_gears      330_transformers&power_supplies  380_signaling&operating  391_consumers&actuators  395_electronics&semiconductors  500_home_installation
110_network_supplies   120_cables&wiring  140_connectors&plugs           310_relays_contactors&contacts  340_converters&inverters         390_sensors&instruments  392_generators&sources   450_high_voltage                qet_directory

=> all folders
open the folder 100_folio_referencing

01coming_arrow.elmt  02going_arrow.elmt  qet_directory

=> .elmt  => start index

  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

Maybe update pugixml to master?
https://github.com/zeux/pugixml/commits/master/src

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."

23 Reply by De-Backer (edited by De-Backer 2020-07-26 22:00:27)

Re: QElectrotech is freezing / loading time library elements

is the xml of the Elements and the project documented?

i would like to find out if we will run into problems with QT 6.
in connection with

As it is mentioned above, SAX classes will be deprecated soon, which means that QDomDocument cannot use them anymore.

https://www.qt.io/blog/parsing-xml-with … s-for-qt-6

grep QDomDocument *
grep: build-aux: Is a directory
grep: dev_doc: Is a directory
grep: doc: Is a directory
grep: docs: Is a directory
grep: elements: Is a directory
grep: examples: Is a directory
grep: ico: Is a directory
grep: lang: Is a directory
grep: man: Is a directory
grep: misc: Is a directory
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_doc, const QString &amp;filepath, QString *error_message=nullptr)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_doc, QFile *file, QString *error_message=nullptr)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;parent_document, const QPen &amp;pen)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;parent_document, const QBrush &amp;brush)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;document, const QDir &amp;dir, const QString &amp;rename=QString())</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;document, QFile &amp;file, const QString &amp;rename=QString())</arglist>
QElectroTech.tag:      <arglist>(const QDomDocument &amp;xml_document, const QString &amp;file_path, QString *error_message=nullptr)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;document, const QString &amp;tag_name, const QString &amp;value)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;parent_document, const QMargins &amp;margins)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;parent_document, const QAbstractItemModel *model, QHash&lt; int, QList&lt; int &gt;&gt; horizontal_section_role, QHash&lt; int, QList&lt; int &gt;&gt; vertical_section_role)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;, QHash&lt; Terminal *, int &gt; &amp;) const</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const =0</arglist>
QElectroTech.tag:      <type>QDomDocument</type>
QElectroTech.tag:      <arglist>(QDomDocument &amp;, QPointF=QPointF(), bool=true, DiagramContent *=nullptr)</arglist>
QElectroTech.tag:      <arglist>(QHash&lt; QString, QStringList &gt; *, QDomElement *, const QString &amp;, const QString &amp;, QDomDocument *)</arglist>
QElectroTech.tag:      <type>QDomDocument</type>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;dom_doc) const override</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;, QHash&lt; Terminal *, int &gt; &amp;) const</arglist>
QElectroTech.tag:      <type>virtual const QDomDocument</type>
QElectroTech.tag:      <arglist>(const QDomDocument &amp;)</arglist>
QElectroTech.tag:      <arglist>(const QDomDocument &amp;, const QPointF &amp;=QPointF(), bool=true, ElementContent *=nullptr)</arglist>
QElectroTech.tag:      <arglist>(const QDomDocument &amp;)</arglist>
QElectroTech.tag:      <arglist>(const QDomDocument &amp;)</arglist>
QElectroTech.tag:      <arglist>(const QDomDocument &amp;xml_document) const</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;dom_document) const</arglist>
QElectroTech.tag:      <arglist>(const QDomDocument &amp;, const QPointF &amp;)</arglist>
QElectroTech.tag:      <arglist>(const QDomDocument &amp;)</arglist>
QElectroTech.tag:      <type>QDomDocument</type>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const override</arglist>
QElectroTech.tag:      <type>QDomDocument</type>
QElectroTech.tag:      <arglist>(QDomDocument &amp;, const QHash&lt; QString, QString &gt; &amp;=QHash&lt; QString, QString &gt;()) const</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;, const QString &amp;)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const override</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;dom_doc) const override</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const override</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const override</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const override</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const override</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const override</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const override</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;document) const</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_document) const =0</arglist>
QElectroTech.tag:      <arglist>(const QDomDocument &amp;)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;document) const</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;dom_document) const</arglist>
QElectroTech.tag:      <type>QDomDocument</type>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_project)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_project)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_project)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_project)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_project)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;document) const</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;document, const QString &amp;tag_name=QString(&quot;sequentialNumbers&quot;)) const</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;) const</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_element) const override</arglist>
QElectroTech.tag:      <type>QDomDocument</type>
QElectroTech.tag:      <type>QDomDocument</type>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_document) const override</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_doc, const QString &amp;filepath, QString *error_message=nullptr)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;xml_doc, QFile *file, QString *error_message=nullptr)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;parent_document, const QPen &amp;pen)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;parent_document, const QBrush &amp;brush)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;document, const QDir &amp;dir, const QString &amp;rename=QString())</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;document, QFile &amp;file, const QString &amp;rename=QString())</arglist>
QElectroTech.tag:      <arglist>(const QDomDocument &amp;xml_document, const QString &amp;file_path, QString *error_message=nullptr)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;document, const QString &amp;tag_name, const QString &amp;value)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;parent_document, const QMargins &amp;margins)</arglist>
QElectroTech.tag:      <arglist>(QDomDocument &amp;parent_document, const QAbstractItemModel *model, QHash&lt; int, QList&lt; int &gt;&gt; horizontal_section_role, QHash&lt; int, QList&lt; int &gt;&gt; vertical_section_role)</arglist>

  • scorpio810
  • scorpio810
  • QElectroTech Team Manager, Developer, Packager
  • Offline

Re: QElectrotech is freezing / loading time library elements

Very very old documentation :
https://qelectrotech.org/wiki_new/doc/x … t_elements
https://qelectrotech.org/wiki_new/doc/x … s_elements

"Le jour où tu découvres le Libre, tu sais que tu ne pourras jamais plus revenir en arrière..."

25 Reply by De-Backer (edited by De-Backer 2020-07-26 23:06:46)

Re: QElectrotech is freezing / loading time library elements

OK thanks, I'm adding it to the todo list.

....
the wiki is very difficult for me to adapt for version 0.8 it is mainly the language (no english).
eg suppose I have to create a new page (well I don't know how to do that)


I will first create the doc on my PC and then I will forward it there to you.
I think that can work..

it will be plain .txt text.