We use scopes.

the whole scoping will be different

Indeed, but a 30-40 line mapping table would get one pretty far... so yes... to use another tokenizer one would have to remap it's named scopes into HLJS named scopes, but that's the easy part AFAIC.

I think you misread "markup" as "markdown".

Ah, sorry I did. If your interest had been in Markdown I was going to point you to some interesting work. :-)

that way you get IDE syntax highlighting,

That would certainly be doable with the existing suite, though I'm not convinced it's a big win. Often the tests include incorrect syntax also, in which case editor highlighting can sometimes be a nuisance.

I'm changing the implementation I'm working on

I'll be curious to see what you propose then...

We've actually been curious about the opposite direction - allowing big editor tokenization to be plugged into us - allowing our themes/rendering engine, but super high fidelity highlighting... of course the price is much, much larger JS payloads... and so far no one seems super interested in that work. https://github.com/highlightjs/highlight.js/issues/3621

Sorry, this usage is out-of-scope and an entirely different usage than what HLJS is intended for. We're not that interested in the live code editor space - lots of solutions there already. That said, you'd be more than welcome to piggy back our grammars and build your own incremental tokenizer on top, but I wouldn't say that's a trivial undertaking. Of course the underlying parser code is all there for forking as well, but I wouldn't say it's super clean - it's never been FULLY refactored since I came onboard.

Worth noting we do not highlight nearly as well as big editors with large (and complete) grammars... we try to exist in the "smart pattern matching" space rather than the "fully parse and understand this language" space... that results in much, much smaller grammars, but often less fidelity. This becomes obvious in more complex grammars with a lot of nuance/context like TSX or Markdown, etc...

Show us your raw HTML.

I'm not sure what this is referring to.

I mean suspend. Where you hit a key but we don't exert it until we know what you're doing - waiting to see if it might be part of a macro that doesn't actually include that key - so we don't "False" press it.

are often out of order, and almost always come out prior to the transformed EV_KEY events.

Yes, very strange.

Wait, I feel like this should be a problem with long macros

Exactly - and also surprising.

Can syn events from a keyboard be easily distinguished from syn events from a pointing device?

Events originate from a specific device. But in combo devices there isn't that delineation.

let the real keyboard send the syn events, allow them through, and not to bother with any artificial ones

No because we sometimes generate events that aren't in line with the keyboard... like if you hit CMD now, but we hit CMD a second later...(with hibernate) so we need the SYN now, not 1 second ago...

Well, it did work ... Now I'm getting identifier 'global' undefined.

So sounds like it's STILL working... that error is a different error entirely... one thing at a time. :-).

Yeah, that is the more "correct" approach IMHO, but really you'd also need some sort of state machine to account for KB + mouse devices (or running a separate mouse thru)... because you have events that are not EV_KEY events (like movement)... so if someone was using a keyboard/mouse we want to steal all their "keys" (and mouse buttons are keys, yes) and so we'd handle SYN for them, but since we pass thru the movement - we'd also need to pass thru SYN events (but only for movement!).

If we failed to do that you could only move the mouse if you hit a key on the keyboard soon after, or some ridiculousness like that... so really I think in a perfect world:

• we always generate artificial-SYN
• we sometimes pass thru natural-SYN, but only when necessary for non-key events

Right now we're doing both which, means sometimes we double up, which evidently some systems don't like.

I assume it's queuing until the "untransformed" syns come thru directly from the real keyboard... which flushes the queue. Why that works so much better than flushing it explicitly after each keypress is what remains a mystery.

For reference, apparently syn() injects a SYN_REPORT, not an EV_SYN event

Same thing, well technically EV_SYN is the whole category... but SYN_REPORT is the only event I'm aware of being used in actual practice.

Events queued by :func:write() will be fired.

This is likely why we were still calling it, if we never called it (and also didn't randomly pass it thru) then any events queued would never be "fired" (and as you've seen that's a no go). So perhaps the secret sauce is in exactly when SYN is exerted?

If possible, events will be merged into an 'atomic' event.

I'd love to know more about this...

I don't know why they exist, but can you not just undefine them?

Yeah, maybe old is a better argument. :-)

This probably isn't what you meant, but ... results seem... unequivocally bad.

Exactly what I meant, and kind of what I expected... so strange though.

Out of curiosity, is there a reason you filed this issue as "improve the markdown harness" vs "improve test harnesses"? Do you have some special interest in markdown over the other grammars?

There is a sweet spot where "a reasonable number of tests" is way better than 1 huge test or 100 small tests. Esp when it comes to finding "where do I add/find this test"... it sounds like you want very small tests. I don't see any advantage to that (for these types of tests).

Hence for more complex languages "strings" is a reasonable level of granularity... there might be multiple types, quoted, unquoted, heredocs, etc... but all of them under strings. For a language with only one type of string, "hello" this would be overkill to have a single file test.

smaller tests helps pinpointing what exactly is broken

I'll simply disagree. The brokeness is always in the Highlight.js grammar/engine... whether the test is 100 files or 1 file doesn't help pinpoint what is broken. And if you mean just literally "[pinpoint] what test is failing" I think our test suite already does a reasonable enough job showing that when tests fail.

and the exercise of writing them can be helpful for finding edge cases.

I don't know that this is exclusively true of small tests...

I find it helpful to see all related test/syntax cases on a single screen to more easily make sure we've covered them all.

Maybe use a union regex?

/^(no-?highlight|mermaid)$/i

What harm are the warnings?

@joshgoebel

EDIT: Solution found later in the thread, I think.

EDIT 2: Created PR #105 with a suggested way to deal with this.

WM_CLASS always seems to be a string, but in some apps WM_NAME can be "(STRING)", or "(UTF8_STRING)" or even "(COMPOUND_TEXT)", which leads to this error from my isWindow() function if it tries to do a case insensitive match:

AttributeError: 'bytes' object has no attribute 'casefold'

WM_CLASS(STRING) = "org.gnome.Nautilus", "Org.gnome.Nautilus"
WM_NAME(STRING) = "Kinto Config Files"

WM_CLASS(STRING) = "org.gnome.Nautilus", "Org.gnome.Nautilus"
WM_NAME(STRING) = "Create Archive"

WM_CLASS(STRING) = "pcloud", "pcloud"
WM_NAME(UTF8_STRING) = "pCloud Drive"

WM_CLASS(STRING) = "org.gnome.Nautilus", "Org.gnome.Nautilus"
WM_NAME(STRING) = "kinto-keyszer.py Properties"

WM_CLASS(STRING) = "org.gnome.Nautilus", "Org.gnome.Nautilus"
WM_NAME(STRING) = "Kinto Config Files"

WM_CLASS(STRING) = "transmission-gtk", "Transmission-gtk"
WM_NAME(STRING) = "Transmission"

WM_CLASS(STRING) = "transmission-gtk", "Transmission-gtk"
WM_NAME(STRING) = "Transmission"

WM_CLASS(STRING) = "pcloud", "pcloud"
WM_NAME(UTF8_STRING) = "pCloud Drive"

WM_CLASS(STRING) = "org.gnome.Nautilus", "Org.gnome.Nautilus"
WM_NAME(STRING) = "Kinto_macOS_Option_Special_Keys"

WM_CLASS(STRING) = "org.gnome.Nautilus", "Org.gnome.Nautilus"
WM_NAME(COMPOUND_TEXT) = "Option-key-cha…BC-Extended.pdf Properties"

Fix for WM_NAME(COMPOUND_TEXT) issue (#105)

• Fix for WM_NAME(COMPOUND_TEXT) issue

As outlined in issue #104, sometimes the "legacy" WM_NAME window attribute returns a result encoded as (COMPOUND_TEXT), which results in window.get_wm_name() returning an empty "bytes" object rather than the correct window title string.

An alternate attribute called _NET_WM_NAME provides the same window title, but in UTF-8 string format that will work properly with regex matching.

Should resolve issue #104.

• Use _NET_WM_NAME instead of WM_NAME

As outlined in issue #104, sometimes the "legacy" WM_NAME window attribute returns a result encoded as (COMPOUND_TEXT), which results in window.get_wm_name() returning an empty "bytes" object rather than the correct window title string.

An alternate attribute called _NET_WM_NAME provides the same window title, but in UTF-8 string format that will work properly with regex matching.

Should resolve issue #104.

Changes

Checklist

• [ ] Added tests (if necessary)
• [ ] Updated docs or README...

I personally don't think we should support transpiled languages... gotta draw the line somewhere.

Are you still sending the original syncs thru the pipeline if you add logging to send_event?

Might leave this open a while (for testing)... it's weird to think we'd need no ACKS... and with this removed the only acks we are getting are going to be original ones that are passed thru (immediately) from the real keyboard (since they are unmappable events)... the proper solution here might be to rewrite the input/modify layer such that it can understands entire key sequences - not just individual events...

IE we have:

• key
• key scan
• ack

So we'd need to understand and rewrite them all.. after this patch we're just remapping key events and passing all scan and ack events straight thru with 0 processing. Amazing that seems to work so well.

I wonder what would happen if we just removed ALL the acks.

It's saying we need to update setuptools... because the old version has a vulnerability...

Uncertain, but for testing purposes everything necessary is in the current build. They key is simply named __emitter vs emitter to denote it's privateness.

FYI: The new website will have this.

I could imagine that, sure, but I'm not sure what the big win would be? I actually think the broad "category" names we already use are pretty descriptive at the "grouping of tests" level (which is what these files typically are - or what we'd prefer them to be)... such as say functions, strings, types, etc...

What is the advantage of taking that filename and ALSO moving it into the first line of the file?

Behavior seems consistent here in latest branch.