diff options
Diffstat (limited to 'src/ext_depends')
| -rw-r--r-- | src/ext_depends/arsd/cgi.d | 1185 | ||||
| -rw-r--r-- | src/ext_depends/arsd/core.d | 6106 |
2 files changed, 7022 insertions, 269 deletions
diff --git a/src/ext_depends/arsd/cgi.d b/src/ext_depends/arsd/cgi.d index 9189052..844a411 100644 --- a/src/ext_depends/arsd/cgi.d +++ b/src/ext_depends/arsd/cgi.d @@ -54,7 +54,7 @@ void main() { */ /++ - Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. + Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. Offers both lower- and higher- level api options among other common (optional) things like websocket and event source serving support, session management, and job scheduling. --- import arsd.cgi; @@ -74,6 +74,30 @@ void main() { mixin GenericMain!hello; --- + Or: + --- + import arsd.cgi; + + class MyApi : WebObject { + @UrlName("") + string hello(string name = null) { + if(name is null) + return "Hello, world!"; + else + return "Hello, " ~ name; + } + } + mixin DispatcherMain!( + "/".serveApi!MyApi + ); + --- + + $(NOTE + Please note that using the higher-level api will add a dependency on arsd.dom and arsd.jsvar to your application. + If you use `dmd -i` or `ldc2 -i` to build, it will just work, but with dub, you will have do `dub add arsd-official:jsvar` + and `dub add arsd-official:dom` yourself. + ) + Test on console (works in any interface mode): $(CONSOLE $ ./cgi_hello GET / name=whatever @@ -85,10 +109,12 @@ void main() { # now you can go to http://localhost:8080/?name=whatever ) - Please note: the default port for http is 8085 and for cgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to hard code your own with [RequestServer]. + Please note: the default port for http is 8085 and for scgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to code your own with [RequestServer], however. - Compile_versions: + Build_Configurations: + + cgi.d tries to be flexible to meet your needs. It is possible to configure it both at runtime (by writing your own `main` function and constructing a [RequestServer] object) or at compile time using the `version` switch to the compiler or a dub `subConfiguration`. If you are using `dub`, use: @@ -105,11 +131,11 @@ void main() { to change versions. The possible options for `VALUE_HERE` are: $(LIST - * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default. - * `cgi` for traditional cgi binaries. - * `fastcgi` for FastCGI builds. - * `scgi` for SCGI builds. - * `stdio_http` for speaking raw http over stdin and stdout. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. + * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default for dub builds. You can run the program then connect directly to it from your browser. + * `cgi` for traditional cgi binaries. These are run by an outside web server as-needed to handle requests. + * `fastcgi` for FastCGI builds. FastCGI is managed from an outside helper, there's one built into Microsoft IIS, Apache httpd, and Lighttpd, and a generic program you can use with nginx called `spawn-fcgi`. If you don't already know how to use it, I suggest you use one of the other modes. + * `scgi` for SCGI builds. SCGI is a simplified form of FastCGI, where you run the server as an application service which is proxied by your outside webserver. + * `stdio_http` for speaking raw http over stdin and stdout. This is made for systemd services. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. ) With dmd, use: @@ -127,7 +153,8 @@ void main() { - A FastCGI executable will be generated. * - `-version=scgi` - A SCGI (SimpleCGI) executable will be generated. - + * - `-version=embedded_httpd_hybrid` + - A HTTP server that uses a combination of processes, threads, and fibers to better handle large numbers of idle connections. Recommended if you are going to serve websockets in a non-local application. * - `-version=embedded_httpd_threads` - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) * - `-version=embedded_httpd_processes` @@ -141,7 +168,7 @@ void main() { + (can be used together with others) * - `-version=cgi_with_websocket` - - The CGI class has websocket server support. + - The CGI class has websocket server support. (This is on by default now.) * - `-version=with_openssl` - not currently used @@ -151,7 +178,7 @@ void main() { - The session will be provided in a separate process, provided by cgi.d. ) - Compile_and_run: + For example, For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. @@ -161,77 +188,24 @@ void main() { For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. - You can also simulate a request by passing parameters on the command line, like: + Simulating_requests: + + If you are using one of the [GenericMain] or [DispatcherMain] mixins, or main with your own call to [RequestServer.trySimulatedRequest], you can simulate requests from your command-ine shell. Call the program like this: $(CONSOLE ./yourprogram GET / name=adr ) - And it will print the result to stdout. + And it will print the result to stdout instead of running a server, regardless of build more.. CGI_Setup_tips: - On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file. + On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file to set a particular file to be run through the cgi program. Note that all "subdirectories" of it also run the program; if you configure `/foo` to be a cgi script, then going to `/foo/bar` will call your cgi handler function with `cgi.pathInfo == "/bar"`. - Integration_tips: + Overview_Of_Basic_Concepts: - cgi.d works well with dom.d for generating html. You may also use web.d for other utilities and automatic api wrapping. + cgi.d offers both lower-level handler apis as well as higher-level auto-dispatcher apis. For a lower-level handler function, you'll probably want to review the following functions: - dom.d usage: - - --- - import arsd.cgi; - import arsd.dom; - - void hello_dom(Cgi cgi) { - auto document = new Document(); - - static import std.file; - // parse the file in strict mode, requiring it to be well-formed UTF-8 XHTML - // (You'll appreciate this if you've ever had to deal with a missing </div> - // or something in a php or erb template before that would randomly mess up - // the output in your browser. Just check it and throw an exception early!) - // - // You could also hard-code a template or load one at compile time with an - // import expression, but you might appreciate making it a regular file - // because that means it can be more easily edited by the frontend team and - // they can see their changes without needing to recompile the program. - // - // Note on CTFE: if you do choose to load a static file at compile time, - // you *can* parse it in CTFE using enum, which will cause it to throw at - // compile time, which is kinda cool too. Be careful in modifying that document, - // though, as it will be a static instance. You might want to clone on on demand, - // or perhaps modify it lazily as you print it out. (Try element.tree, it returns - // a range of elements which you could send through std.algorithm functions. But - // since my selector implementation doesn't work on that level yet, you'll find that - // harder to use. Of course, you could make a static list of matching elements and - // then use a simple e is e2 predicate... :) ) - document.parseUtf8(std.file.read("your_template.html"), true, true); - - // fill in data using DOM functions, so placing it is in the hands of HTML - // and it will be properly encoded as text too. - // - // Plain html templates can't run server side logic, but I think that's a - // good thing - it keeps them simple. You may choose to extend the html, - // but I think it is best to try to stick to standard elements and fill them - // in with requested data with IDs or class names. A further benefit of - // this is the designer can also highlight data based on sources in the CSS. - // - // However, all of dom.d is available, so you can format your data however - // you like. You can do partial templates with innerHTML too, or perhaps better, - // injecting cloned nodes from a partial document. - // - // There's a lot of possibilities. - document["#name"].innerText = cgi.request("name", "default name"); - - // send the document to the browser. The second argument to `cgi.write` - // indicates that this is all the data at once, enabling a few small - // optimizations. - cgi.write(document.toString(), true); - } - --- - - Concepts: Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) @@ -245,12 +219,175 @@ void main() { Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] - Overriding behavior: [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] + Websockets: [Websocket], [websocketRequested], [acceptWebsocket]. For websockets, use the `embedded_httpd_hybrid` build mode for best results, because it is optimized for handling large numbers of idle connections compared to the other build modes. + + Overriding behavior for special cases streaming input data: see the virtual functions [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] + + A basic program using the lower-level api might look like: + + --- + import arsd.cgi; + + // you write a request handler which always takes a Cgi object + void handler(Cgi cgi) { + /+ + when the user goes to your site, suppose you are being hosted at http://example.com/yourapp + + If the user goes to http://example.com/yourapp/test?name=value + then the url will be parsed out into the following pieces: + + cgi.pathInfo == "/test". This is everything after yourapp's name. (If you are doing an embedded http server, your app's name is blank, so pathInfo will be the whole path of the url.) + + cgi.scriptName == "yourapp". With an embedded http server, this will be blank. + + cgi.host == "example.com" + + cgi.https == false + + cgi.queryString == "name=value" (there's also cgi.search, which will be "?name=value", including the ?) + + The query string is further parsed into the `get` and `getArray` members, so: + + cgi.get == ["name": "value"], meaning you can do `cgi.get["name"] == "value"` + + And + + cgi.getArray == ["name": ["value"]]. + + Why is there both `get` and `getArray`? The standard allows names to be repeated. This can be very useful, + it is how http forms naturally pass multiple items like a set of checkboxes. So `getArray` is the complete data + if you need it. But since so often you only care about one value, the `get` member provides more convenient access. + + We can use these members to process the request and build link urls. Other info from the request are in other members, we'll look at them later. + +/ + switch(cgi.pathInfo) { + // the home page will be a small html form that can set a cookie. + case "/": + cgi.write(`<!DOCTYPE html> + <html> + <body> + <form method="POST" action="set-cookie"> + <label>Your name: <input type="text" name="name" /></label> + <input type="submit" value="Submit" /> + </form> + </body> + </html> + `, true); // the , true tells it that this is the one, complete response i want to send, allowing some optimizations. + break; + // POSTing to this will set a cookie with our submitted name + case "/set-cookie": + // HTTP has a number of request methods (also called "verbs") to tell + // what you should do with the given resource. + // The most common are GET and POST, the ones used in html forms. + // You can check which one was used with the `cgi.requestMethod` property. + if(cgi.requestMethod == Cgi.RequestMethod.POST) { + + // headers like redirections need to be set before we call `write` + cgi.setResponseLocation("read-cookie"); + + // just like how url params go into cgi.get/getArray, form data submitted in a POST + // body go to cgi.post/postArray. Please note that a POST request can also have get + // params in addition to post params. + // + // There's also a convenience function `cgi.request("name")` which checks post first, + // then get if it isn't found there, and then returns a default value if it is in neither. + if("name" in cgi.post) { + // we can set cookies with a method too + // again, cookies need to be set before calling `cgi.write`, since they + // are a kind of header. + cgi.setCookie("name" , cgi.post["name"]); + } + + // the user will probably never see this, since the response location + // is an automatic redirect, but it is still best to say something anyway + cgi.write("Redirecting you to see the cookie...", true); + } else { + // you can write out response codes and headers + // as well as response bodies + // + // But always check the cgi docs before using the generic + // `header` method - if there is a specific method for your + // header, use it before resorting to the generic one to avoid + // a header value from being sent twice. + cgi.setResponseLocation("405 Method Not Allowed"); + // there is no special accept member, so you can use the generic header function + cgi.header("Accept: POST"); + // but content type does have a method, so prefer to use it: + cgi.setResponseContentType("text/plain"); + + // all the headers are buffered, and will be sent upon the first body + // write. you can actually modify some of them before sending if need be. + cgi.write("You must use the POST http verb on this resource.", true); + } + break; + // and GETting this will read the cookie back out + case "/read-cookie": + // I did NOT pass `,true` here because this is writing a partial response. + // It is possible to stream data to the user in chunks by writing partial + // responses the calling `cgi.flush();` to send the partial response immediately. + // normally, you'd only send partial chunks if you have to - it is better to build + // a response as a whole and send it as a whole whenever possible - but here I want + // to demo that you can. + cgi.write("Hello, "); + if("name" in cgi.cookies) { + import arsd.dom; // dom.d provides a lot of helpers for html + // since the cookie is set, we need to write it out properly to + // avoid cross-site scripting attacks. + // + // Getting this stuff right automatically is a benefit of using the higher + // level apis, but this demo is to show the fundamental building blocks, so + // we're responsible to take care of it. + cgi.write(htmlEntitiesEncode(cgi.cookies["name"])); + } else { + cgi.write("friend"); + } + + // note that I never called cgi.setResponseContentType, since the default is text/html. + // it doesn't hurt to do it explicitly though, just remember to do it before any cgi.write + // calls. + break; + default: + // no path matched + cgi.setResponseStatus("404 Not Found"); + cgi.write("Resource not found.", true); + } + } + + // and this adds the boilerplate to set up a server according to the + // compile version configuration and call your handler as requests come in + mixin GenericMain!handler; // the `handler` here is the name of your function + --- + + Even if you plan to always use the higher-level apis, I still recommend you at least familiarize yourself with the lower level functions, since they provide the lightest weight, most flexible options to get down to business if you ever need them. + + In the lower-level api, the [Cgi] object represents your HTTP transaction. It has functions to describe the request and for you to send your response. It leaves the details of how you o it up to you. The general guideline though is to avoid depending any variables outside your handler function, since there's no guarantee they will survive to another handler. You can use global vars as a lazy initialized cache, but you should always be ready in case it is empty. (One exception: if you use `-version=embedded_httpd_threads -version=cgi_no_fork`, then you can rely on it more, but you should still really write things assuming your function won't have anything survive beyond its return for max scalability and compatibility.) + + A basic program using the higher-level apis might look like: - Installing: Apache, IIS, CGI, FastCGI, SCGI, embedded HTTPD (not recommended for production use) + --- + /+ + import arsd.cgi; + + struct LoginData { + string currentUser; + } + + class AppClass : WebObject { + string foo() {} + } + + mixin DispatcherMain!( + "/assets/.serveStaticFileDirectory("assets/", true), // serve the files in the assets subdirectory + "/".serveApi!AppClass, + "/thing/".serveRestObject, + ); + +/ + --- Guide_for_PHP_users: - If you are coming from PHP, here's a quick guide to help you get started: + (Please note: I wrote this section in 2008. A lot of PHP hosts still ran 4.x back then, so it was common to avoid using classes - introduced in php 5 - to maintain compatibility! If you're coming from php more recently, this may not be relevant anymore, but still might help you.) + + If you are coming from old-style PHP, here's a quick guide to help you get started: $(SIDE_BY_SIDE $(COLUMN @@ -326,27 +463,113 @@ void main() { See_Also: - You may also want to see [arsd.dom], [arsd.web], and [arsd.html] for more code for making - web applications. + You may also want to see [arsd.dom], [arsd.webtemplate], and maybe some functions from my old [arsd.html] for more code for making + web applications. dom and webtemplate are used by the higher-level api here in cgi.d. For working with json, try [arsd.jsvar]. [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in accessing databases. - If you are looking to access a web application via HTTP, try [std.net.curl], [arsd.curl], or [arsd.http2]. + If you are looking to access a web application via HTTP, try [arsd.http2]. Copyright: - cgi.d copyright 2008-2021, Adam D. Ruppe. Provided under the Boost Software License. + cgi.d copyright 2008-2023, Adam D. Ruppe. Provided under the Boost Software License. Yes, this file is old, and yes, it is still actively maintained and used. + + History: + An import of `arsd.core` was added on March 21, 2023 (dub v11.0). Prior to this, the module's default configuration was completely stand-alone. You must now include the `core.d` file in your builds with `cgi.d`. + + This change is primarily to integrate the event loops across the library, allowing you to more easily use cgi.d along with my other libraries like simpledisplay and http2.d. Previously, you'd have to run separate helper threads. Now, they can all automatically work together. +/ module arsd.cgi; +static import arsd.core; +version(Posix) +import arsd.core : makeNonBlocking; + +// FIXME: Nullable!T can be a checkbox that enables/disables the T on the automatic form +// and a SumType!(T, R) can be a radio box to pick between T and R to disclose the extra boxes on the automatic form + +/++ + This micro-example uses the [dispatcher] api to act as a simple http file server, serving files found in the current directory and its children. ++/ +version(Demo) +unittest { + import arsd.cgi; + + mixin DispatcherMain!( + "/".serveStaticFileDirectory(null, true) + ); +} + +/++ + Same as the previous example, but written out long-form without the use of [DispatcherMain] nor [GenericMain]. ++/ +version(Demo) +unittest { + import arsd.cgi; + + void requestHandler(Cgi cgi) { + cgi.dispatcher!( + "/".serveStaticFileDirectory(null, true) + ); + } + + // mixin GenericMain!requestHandler would add this function: + void main(string[] args) { + // this is all the content of [cgiMainImpl] which you can also call + + // cgi.d embeds a few add on functions like real time event forwarders + // and session servers it can run in other processes. this spawns them, if needed. + if(tryAddonServers(args)) + return; + + // cgi.d allows you to easily simulate http requests from the command line, + // without actually starting a server. this function will do that. + if(trySimulatedRequest!(requestHandler, Cgi)(args)) + return; + + RequestServer server; + // you can change the default port here if you like + // server.listeningPort = 9000; + + // then call this to let the command line args override your default + server.configureFromCommandLine(args); + + // here is where you could print out the listeningPort to the user if you wanted + + // and serve the request(s) according to the compile configuration + server.serve!(requestHandler)(); + + // or you could explicitly choose a serve mode like this: + // server.serveEmbeddedHttp!requestHandler(); + } +} + +/++ + cgi.d has built-in testing helpers too. These will provide mock requests and mock sessions that + otherwise run through the rest of the internal mechanisms to call your functions without actually + spinning up a server. ++/ version(Demo) unittest { + import arsd.cgi; + + void requestHandler(Cgi cgi) { + + } + + // D doesn't let me embed a unittest inside an example unittest + // so this is a function, but you can do it however in your real program + /* unittest */ void runTests() { + auto tester = new CgiTester(&requestHandler); + auto response = tester.GET("/"); + assert(response.code == 200); + } } static import std.file; @@ -360,9 +583,7 @@ version(Posix) { } else version(minimal) { } else { - version(GNU) { - // GDC doesn't support static foreach so I had to cheat on it :( - } else version(FreeBSD) { + version(FreeBSD) { // I never implemented the fancy stuff there either } else { version=with_breaking_cgi_features; @@ -381,6 +602,7 @@ version(Windows) { } } +// FIXME: can use the arsd.core function now but it is trivial anyway tbh void cloexec(int fd) { version(Posix) { import core.sys.posix.fcntl; @@ -402,6 +624,11 @@ version(embedded_httpd_hybrid) { version=cgi_use_fiber; } +version(cgi_use_fork) + enum cgi_use_fork_default = true; +else + enum cgi_use_fork_default = false; + // the servers must know about the connections to talk to them; the interfaces are vital version(with_addon_servers) version=with_addon_servers_connections; @@ -952,7 +1179,7 @@ class Cgi { { import core.runtime; auto sfn = getenv("SCRIPT_FILENAME"); - scriptFileName = sfn.length ? sfn : Runtime.args[0]; + scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); } bool iis = false; @@ -1031,7 +1258,8 @@ class Cgi { // to be slow if they did that. The spec says it is always there though. // And it has worked reliably for me all year in the live environment, // but some servers might be different. - auto contentLength = to!size_t(getenv("CONTENT_LENGTH")); + auto cls = getenv("CONTENT_LENGTH"); + auto contentLength = to!size_t(cls.length ? cls : "0"); immutable originalContentLength = contentLength; if(contentLength) { @@ -1745,7 +1973,7 @@ class Cgi { { import core.runtime; - scriptFileName = Runtime.args[0]; + scriptFileName = Runtime.args.length ? Runtime.args[0] : null; } @@ -1860,6 +2088,8 @@ class Cgi { // FIXME: if size is > max content length it should // also fail at this point. _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); + + // FIXME: let the user write out 103 early hints too } } // else @@ -2196,6 +2426,17 @@ class Cgi { customHeaders ~= h; } + /++ + I named the original function `header` after PHP, but this pattern more fits + the rest of the Cgi object. + + Either name are allowed. + + History: + Alias added June 17, 2022. + +/ + alias setResponseHeader = header; + private string[] customHeaders; private bool websocketMode; @@ -2398,7 +2639,7 @@ class Cgi { return; // don't double close if(!outputtedResponseData) - write("", false, false); + write("", true, false); // writing auto buffered data if(requestMethod != RequestMethod.HEAD && autoBuffer) { @@ -2523,6 +2764,23 @@ class Cgi { version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. immutable(ubyte)[] originalPostData; + /++ + This holds the posted body data if it has not been parsed into [post] and [postArray]. + + It is intended to be used for JSON and XML request content types, but also may be used + for other content types your application can handle. But it will NOT be populated + for content types application/x-www-form-urlencoded or multipart/form-data, since those are + parsed into the post and postArray members. + + Remember that anything beyond your `maxContentLength` param when setting up [GenericMain], etc., + will be discarded to the client with an error. This helps keep this array from being exploded in size + and consuming all your server's memory (though it may still be possible to eat excess ram from a concurrent + client in certain build modes.) + + History: + Added January 5, 2021 + Documented February 21, 2023 (dub v11.0) + +/ public immutable string postBody; alias postJson = postBody; // old name @@ -2893,7 +3151,10 @@ struct Uri { host = authority; } else { host = authority[0 .. idx2]; - port = to!int(authority[idx2 + 1 .. $]); + if(idx2 + 1 < authority.length) + port = to!int(authority[idx2 + 1 .. $]); + else + port = 0; } } @@ -2969,6 +3230,8 @@ struct Uri { /// Browsers use a function like this to figure out links in html. Uri basedOn(in Uri baseUrl) const { Uri n = this; // copies + if(n.scheme == "data") + return n; // n.uriInvalidated = true; // make sure we regenerate... // userinfo is not inherited... is this wrong? @@ -3278,7 +3541,11 @@ string toHexUpper(long num) { // the generic mixins -/// Use this instead of writing your own main +/++ + Use this instead of writing your own main + + It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. ++/ mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { mixin CustomCgiMain!(Cgi, fun, maxContentLength); } @@ -3292,6 +3559,23 @@ mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentL Added July 9, 2021 +/ mixin template DispatcherMain(Presenter, DispatcherArgs...) { + /// forwards to [CustomCgiDispatcherMain] with default args + mixin CustomCgiDispatcherMain!(Cgi, defaultMaxContentLength, Presenter, DispatcherArgs); +} + +/// ditto +mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { + class GenericPresenter : WebPresenter!GenericPresenter {} + mixin DispatcherMain!(GenericPresenter, DispatcherArgs); +} + +/++ + Allows for a generic [DispatcherMain] with custom arguments. Note you can use [defaultMaxContentLength] as the second argument if you like. + + History: + Added May 13, 2023 (dub v11.0) ++/ +mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, Presenter, DispatcherArgs...) { /++ Handler to the generated presenter you can use from your objects, etc. +/ @@ -3326,12 +3610,14 @@ mixin template DispatcherMain(Presenter, DispatcherArgs...) { presenter.renderBasicError(cgi, 404); } } - mixin GenericMain!handler; + mixin CustomCgiMain!(CustomCgi, handler, maxContentLength); } -mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { +/// ditto +mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { class GenericPresenter : WebPresenter!GenericPresenter {} - mixin DispatcherMain!(GenericPresenter, DispatcherArgs); + mixin CustomCgiDispatcherMain!(CustomCgi, maxContentLength, GenericPresenter, DispatcherArgs); + } private string simpleHtmlEncode(string s) { @@ -3478,6 +3764,28 @@ struct RequestServer { /// ushort listeningPort = defaultListeningPort(); + /++ + Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the + other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But + if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and + [stop] may not work as well. + + History: + Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` + argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for + compatibility. + +/ + bool useFork = cgi_use_fork_default; + + /++ + Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a + default based on the number of cpus modified by the server mode. + + History: + Added August 12, 2022 (dub v10.9) + +/ + int numberOfThreads = 0; + /// this(string defaultHost, ushort defaultPort) { this.listeningHost = defaultHost; @@ -3509,11 +3817,11 @@ struct RequestServer { foundHost = false; } if(foundUid) { - privDropUserId = to!int(arg); + privilegesDropToUid = to!uid_t(arg); foundUid = false; } if(foundGid) { - privDropGroupId = to!int(arg); + privilegesDropToGid = to!gid_t(arg); foundGid = false; } if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") @@ -3527,7 +3835,37 @@ struct RequestServer { } } - // FIXME: the privDropUserId/group id need to be set in here instead of global + version(Windows) { + private alias uid_t = int; + private alias gid_t = int; + } + + /// user (uid) to drop privileges to + /// 0 … do nothing + uid_t privilegesDropToUid = 0; + /// group (gid) to drop privileges to + /// 0 … do nothing + gid_t privilegesDropToGid = 0; + + private void dropPrivileges() { + version(Posix) { + import core.sys.posix.unistd; + + if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) + throw new Exception("Dropping privileges via setgid() failed."); + + if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) + throw new Exception("Dropping privileges via setuid() failed."); + } + else { + // FIXME: Windows? + //pragma(msg, "Dropping privileges is not implemented for this platform"); + } + + // done, set zero + privilegesDropToGid = 0; + privilegesDropToUid = 0; + } /++ Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders @@ -3556,7 +3894,7 @@ struct RequestServer { bool tcp; void delegate() cleanup; - auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1); + auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); auto connection = socket.accept(); doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); @@ -3603,6 +3941,7 @@ struct RequestServer { If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. +/ void serveEmbeddedHttp(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(ThisFor!fun _this) { + globalStopFlag = false; static if(__traits(isStaticFunction, fun)) alias funToUse = fun; else @@ -3611,7 +3950,7 @@ struct RequestServer { __traits(child, _this, fun)(cgi); else static assert(0, "Not implemented in your compiler version!"); } - auto manager = new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse)); + auto manager = new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); manager.listen(); } @@ -3619,14 +3958,15 @@ struct RequestServer { Runs the embedded SCGI server specifically, regardless of which build configuration you have. +/ void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { - auto manager = new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength)); + globalStopFlag = false; + auto manager = new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); manager.listen(); } /++ Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. - Intended for cases like working from systemd, like discussed here: https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org + Intended for cases like working from systemd, like discussed here: [https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org] History: Added May 29, 2021 @@ -3636,28 +3976,60 @@ struct RequestServer { } /++ - Stops serving after the current requests. + The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't + respond to this flag, the library will force the issue. This determines when and how the issue will be forced. + +/ + enum ForceStop { + /++ + Stops accepting new requests, but lets ones already in the queue start and complete before exiting. + +/ + afterQueuedRequestsComplete, + /++ + Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers + should cooperate and exit gracefully, but if they don't, it will continue waiting for them. + +/ + afterCurrentRequestsComplete, + /++ + Partial response writes will throw an exception, cancelling any streaming response, but complete + writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. + +/ + cancelStreamingRequestsEarly, + /++ + All writes will throw. + +/ + cancelAllRequestsEarly, + /++ + Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). + +/ + forciblyTerminate, + } + + version(embedded_httpd_processes) {} else + /++ + Stops serving after the current requests are completed. Bugs: - Not implemented on version=embedded_httpd_processes, version=fastcgi, or on any operating system aside from Linux at this time. - Try SIGINT there perhaps. + Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid + on Windows however). Only partially implemented on non-Linux posix systems. + + You might also try SIGINT perhaps. - A Windows implementation is planned but not sure about the others. Maybe a posix pipe can be used on other OSes. I do not intend - to implement this for the processes config. + The stopPriority is not yet fully implemented. +/ - version(embedded_httpd_processes) {} else - static void stop() { + static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { globalStopFlag = true; - version(Posix) - if(cancelfd > 0) { - ulong a = 1; - core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); + version(Posix) { + if(cancelfd > 0) { + ulong a = 1; + core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); + } } - version(Windows) - if(iocp) { - foreach(i; 0 .. 16) // FIXME - PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); + version(Windows) { + if(iocp) { + foreach(i; 0 .. 16) // FIXME + PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); + } } } } @@ -3679,28 +4051,6 @@ else private __gshared bool globalStopFlag = false; -private int privDropUserId; -private int privDropGroupId; - -// Added Jan 11, 2021 -private void dropPrivs() { - version(Posix) { - import core.sys.posix.unistd; - - auto userId = privDropUserId; - auto groupId = privDropGroupId; - - if((userId != 0 || groupId != 0) && getuid() == 0) { - if(groupId) - setgid(groupId); - if(userId) - setuid(userId); - } - - } - // FIXME: Windows? -} - version(embedded_httpd_processes) void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { import core.sys.posix.unistd; @@ -3744,7 +4094,7 @@ void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer param close(sock); throw new Exception("listen"); } - dropPrivs(); + params.dropPrivileges(); } version(embedded_httpd_processes_accept_after_fork) {} else { @@ -4082,6 +4432,8 @@ string defaultListeningHost() { /++ This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. + Please note that this may spawn other helper processes that will call `main` again. It does this currently for the timer server and event source server (and the quasi-deprecated web socket server). + Params: fun = Your request handler CustomCgi = a subclass of Cgi, if you wise to customize it further @@ -4089,7 +4441,7 @@ string defaultListeningHost() { args = command-line arguments History: - Documented Sept 26, 2020. + Documented Sept 26, 2020. +/ void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { if(tryAddonServers(args)) @@ -4294,7 +4646,7 @@ extern(Windows) private { alias LPWSAOVERLAPPED = LPOVERLAPPED; /+ - alias LPFN_ACCEPTEX = + alias LPFN_ACCEPTEX = BOOL function( SOCKET sListenSocket, @@ -4437,7 +4789,7 @@ private class PseudoblockingOverlappedSocket : Socket { WSABUF[1] buffer; OVERLAPPED overlapped; - override ptrdiff_t send(const(void)[] buf, SocketFlags flags) @trusted { + override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { overlapped = overlapped.init; buffer[0].len = cast(DWORD) buf.length; buffer[0].buf = cast(CHAR*) buf.ptr; @@ -4452,7 +4804,7 @@ private class PseudoblockingOverlappedSocket : Socket { Fiber.yield(); return lastAnswer; } - override ptrdiff_t receive(void[] buf, SocketFlags flags) @trusted { + override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { overlapped = overlapped.init; buffer[0].len = cast(DWORD) buf.length; buffer[0].buf = cast(CHAR*) buf.ptr; @@ -4473,16 +4825,16 @@ private class PseudoblockingOverlappedSocket : Socket { } // I might go back and implement these for udp things. - override ptrdiff_t receiveFrom(void[] buf, SocketFlags flags, ref Address from) @trusted { + override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { assert(0); } - override ptrdiff_t receiveFrom(void[] buf, SocketFlags flags) @trusted { + override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { assert(0); } - override ptrdiff_t sendTo(const(void)[] buf, SocketFlags flags, Address to) @trusted { + override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { assert(0); } - override ptrdiff_t sendTo(const(void)[] buf, SocketFlags flags) @trusted { + override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { assert(0); } @@ -5108,20 +5460,20 @@ private class FakeSocketForStdin : Socket { private bool closed; - override ptrdiff_t receive(void[] buffer, std.socket.SocketFlags) @trusted { + override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { if(closed) throw new Exception("Closed"); return stdin.rawRead(buffer).length; } - override ptrdiff_t send(const void[] buffer, std.socket.SocketFlags) @trusted { + override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { if(closed) throw new Exception("Closed"); stdout.rawWrite(buffer); return buffer.length; } - override void close() @trusted { + override void close() @trusted scope { (cast(void delegate() @nogc nothrow) &realClose)(); } @@ -5129,7 +5481,7 @@ private class FakeSocketForStdin : Socket { // FIXME } - override void setOption(SocketOptionLevel, SocketOption, void[]) {} + override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} override void setOption(SocketOptionLevel, SocketOption, Duration) {} override @property @trusted Address remoteAddress() { return null; } @@ -5152,9 +5504,13 @@ import core.atomic; /** To use this thing: + --- void handler(Socket s) { do something... } - auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler); + auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); manager.listen(); + --- + + The 4th parameter is optional. I suggest you use BufferedInputRange(connection) to handle the input. As a packet comes in, you will get control. You can just continue; though to fetch more. @@ -5175,7 +5531,8 @@ class ListeningConnectionManager { fd_set read_fds; FD_ZERO(&read_fds); FD_SET(listener.handle, &read_fds); - FD_SET(cancelfd, &read_fds); + if(cancelfd != -1) + FD_SET(cancelfd, &read_fds); auto max = listener.handle > cancelfd ? listener.handle : cancelfd; auto ret = select(max + 1, &read_fds, null, null, null); if(ret == -1) { @@ -5186,7 +5543,7 @@ class ListeningConnectionManager { throw new Exception("wtf select"); } - if(FD_ISSET(cancelfd, &read_fds)) { + if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { return null; } @@ -5194,8 +5551,36 @@ class ListeningConnectionManager { return listener.accept(); return null; - } else - return listener.accept(); // FIXME: check the cancel flag! + } else { + + Socket socket = listener; + + auto check = new SocketSet(); + + keep_looping: + check.reset(); + check.add(socket); + + // just to check the stop flag on a kinda busy loop. i hate this FIXME + auto got = Socket.select(check, null, null, 3.seconds); + if(got > 0) + return listener.accept(); + if(globalStopFlag) + return null; + else + goto keep_looping; + } + } + + int defaultNumberOfThreads() { + import std.parallelism; + version(cgi_use_fiber) { + return totalCPUs * 1 + 1; + } else { + // I times 4 here because there's a good chance some will be blocked on i/o. + return totalCPUs * 4; + } + } void listen() { @@ -5229,11 +5614,12 @@ class ListeningConnectionManager { } } } else { - import std.parallelism; - version(cgi_use_fork) { - //asm { int 3; } - fork(); + if(useFork) { + version(linux) { + //asm { int 3; } + fork(); + } } version(cgi_use_fiber) { @@ -5242,7 +5628,7 @@ class ListeningConnectionManager { listener.accept(); } - WorkerThread[] threads = new WorkerThread[](totalCPUs * 1 + 1); + WorkerThread[] threads = new WorkerThread[](numberOfThreads); foreach(i, ref thread; threads) { thread = new WorkerThread(this, handler, cast(int) i); thread.start(); @@ -5269,8 +5655,7 @@ class ListeningConnectionManager { } else { semaphore = new Semaphore(); - // I times 4 here because there's a good chance some will be blocked on i/o. - ConnectionThread[] threads = new ConnectionThread[](totalCPUs * 4); + ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); foreach(i, ref thread; threads) { thread = new ConnectionThread(this, handler, cast(int) i); thread.start(); @@ -5354,17 +5739,20 @@ class ListeningConnectionManager { private void dg_handler(Socket s) { fhandler(s); } - this(string host, ushort port, void function(Socket) handler) { + this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { fhandler = handler; - this(host, port, &dg_handler); + this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); } - this(string host, ushort port, void delegate(Socket) handler) { + this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { this.handler = handler; + this.useFork = useFork; + this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); - listener = startListening(host, port, tcp, cleanup, 128); + listener = startListening(host, port, tcp, cleanup, 128, dropPrivs); - version(cgi_use_fiber) version(cgi_use_fork) + version(cgi_use_fiber) + if(useFork) listener.blocking = false; // this is the UI control thread and thus gets more priority @@ -5373,9 +5761,12 @@ class ListeningConnectionManager { Socket listener; void delegate(Socket) handler; + + immutable bool useFork; + int numberOfThreads; } -Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue) { +Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { Socket listener; if(host.startsWith("unix:")) { version(Posix) { @@ -5423,7 +5814,8 @@ Socket startListening(string host, ushort port, ref bool tcp, ref void delegate( listener.listen(backQueue); - dropPrivs(); + if (dropPrivs !is null) // can be null, backwards compatibility + dropPrivs(); return listener; } @@ -6573,8 +6965,8 @@ version(Windows) } version(Posix) { + import core.sys.posix.unistd; version(CRuntime_Musl) {} else { - import core.sys.posix.unistd; private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); } } @@ -6758,18 +7150,6 @@ void runSessionServer()() { runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); } -version(Posix) -private void makeNonBlocking(int fd) { - import core.sys.posix.fcntl; - auto flags = fcntl(fd, F_GETFL, 0); - if(flags == -1) - throw new Exception("fcntl get"); - flags |= O_NONBLOCK; - auto s = fcntl(fd, F_SETFL, flags); - if(s == -1) - throw new Exception("fcntl set"); -} - import core.stdc.errno; struct IoOp { @@ -6905,7 +7285,7 @@ interface EventIoServer { } // the sink should buffer it -private void serialize(T)(scope void delegate(ubyte[]) sink, T t) { +private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { static if(is(T == struct)) { foreach(member; __traits(allMembers, T)) serialize(sink, __traits(getMember, t, member)); @@ -6984,7 +7364,7 @@ unittest { }, 56674); ubyte[1000] buffer; int bufferPoint; - void add(ubyte[] b) { + void add(scope ubyte[] b) { buffer[bufferPoint .. bufferPoint + b.length] = b[]; bufferPoint += b.length; } @@ -7052,7 +7432,7 @@ mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. static foreach(idx, member; __traits(derivedMembers, T)) { - static if(__traits(isVirtualFunction, __traits(getMember, T, member))) + static if(__traits(isVirtualMethod, __traits(getMember, T, member))) mixin( q{ std.traits.ReturnType!(__traits(getMember, T, member)) } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) @@ -7087,9 +7467,9 @@ mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) int dataLocation; ubyte[] grab(int sz) { - auto d = got[dataLocation .. dataLocation + sz]; + auto dataLocation1 = dataLocation; dataLocation += sz; - return d; + return got[dataLocation1 .. dataLocation]; } typeof(return) retu; @@ -7145,7 +7525,7 @@ void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(i sw: switch(calledIdx) { foreach(idx, memberName; __traits(derivedMembers, Interface)) - static if(__traits(isVirtualFunction, __traits(getMember, Interface, memberName))) { + static if(__traits(isVirtualMethod, __traits(getMember, Interface, memberName))) { case idx: assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); @@ -7955,7 +8335,7 @@ final class EventSourceServerImplementation : EventSourceServer, EventIoServer { int typeLength; char[32] typeBuffer = 0; int messageLength; - char[2048] messageBuffer = 0; + char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. int _lifetime; char[] message() return { @@ -8228,7 +8608,7 @@ void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoS cloexec(ns); makeNonBlocking(ns); - auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096, &eis.handleLocalConnectionData); + auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); niop.closeHandler = &eis.handleLocalConnectionClose; niop.completeHandler = &eis.handleLocalConnectionComplete; scope(failure) freeIoOp(niop); @@ -8732,6 +9112,9 @@ auto callFromCgi(alias method, T)(T dg, Cgi cgi) { } return false; + } else static if(is(T == enum)) { + *what = to!T(value); + return true; } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { *what = to!T(value); return true; @@ -9156,8 +9539,10 @@ css"; Element htmlContainer() { auto document = new Document(q"html <!DOCTYPE html> -<html> +<html class="no-script"> <head> + <script>document.documentElement.classList.remove("no-script");</script> + <style>.no-script requires-script { display: none; }</style> <title>D Application</title> <link rel="stylesheet" href="style.css" /> </head> @@ -9190,8 +9575,33 @@ html", true, true); } void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { - // FIXME? format? - (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); + switch(format) { + case "html": + (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); + break; + case "json": + import arsd.jsvar; + static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { + var json; + foreach(index, type; Types) { + if(ret.contains == index) + json = ret.payload[index]; + } + } else { + var json = ret; + } + var envelope = json; // var.emptyObject; + /* + envelope.success = true; + envelope.result = json; + envelope.error = null; + */ + cgi.setResponseContentType("application/json"); + cgi.write(envelope.toJson(), true); + break; + default: + cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. + } } /// typeof(null) (which is also used to represent functions returning `void`) do nothing @@ -9231,9 +9641,14 @@ html", true, true); assert(0); } - /// An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort. + /++ + An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort if the filename member is non-null of the FileResource interface. + +/ void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { cgi.setCache(true); // not necessarily true but meh + if(auto fn = ret.filename()) { + cgi.header("Content-Disposition: attachment; filename="~fn~";"); + } cgi.setResponseContentType(ret.contentType); cgi.write(ret.getData(), true); } @@ -9246,6 +9661,21 @@ html", true, true); } /++ + + History: + Added January 23, 2023 (dub v11.0) + +/ + void presentExceptionalReturn(Meta)(Cgi cgi, Throwable t, Meta meta, string format) { + switch(format) { + case "html": + presentExceptionAsHtml(cgi, t, meta); + break; + default: + } + } + + + /++ If you override this, you will need to cast the exception type `t` dynamically, but can then use the template arguments here to refer back to the function. @@ -9253,14 +9683,29 @@ html", true, true); method on the live object. You could, in theory, change arguments and retry, but I provide that information mostly with the expectation that you will use them to make useful forms or richer error messages for the user. + + History: + BREAKING CHANGE on January 23, 2023 (v11.0 ): it previously took an `alias func` and `T dg` to call the function again. + I removed this in favor of a `Meta` param. + + Before: `void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg)` + + After: `void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta)` + + If you used the func for something, move that something into your `methodMeta` template. + + What is the benefit of this change? Somewhat smaller executables and faster builds thanks to more reused functions, together with + enabling an easier implementation of [presentExceptionalReturn]. +/ - void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg) { + void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta) { Form af; + /+ foreach(attr; __traits(getAttributes, func)) { static if(__traits(isSame, attr, AutomaticForm)) { af = createAutomaticFormForFunction!(func)(dg); } } + +/ presentExceptionAsHtmlImpl(cgi, t, af); } @@ -9363,6 +9808,21 @@ html", true, true); auto i = lbl.addChild("input", name); i.attrs.name = name; i.attrs.type = "file"; + } else static if(is(T == enum)) { + Element lbl; + if(displayName !is null) { + lbl = div.addChild("label"); + lbl.addChild("span", displayName, "label-text"); + lbl.appendText(" "); + } else { + lbl = div; + } + auto i = lbl.addChild("select", name); + i.attrs.name = name; + + foreach(memberName; __traits(allMembers, T)) + i.addChild("option", memberName); + } else static if(is(T == struct)) { if(displayName !is null) div.addChild("span", displayName, "label-text"); @@ -9411,18 +9871,6 @@ html", true, true); i.attrs.type = "checkbox"; i.attrs.value = "true"; i.attrs.name = name; - } else static if(is(T == Cgi.UploadedFile)) { - Element lbl; - if(displayName !is null) { - lbl = div.addChild("label"); - lbl.addChild("span", displayName, "label-text"); - lbl.appendText(" "); - } else { - lbl = div; - } - auto i = lbl.addChild("input", name); - i.attrs.name = name; - i.attrs.type = "file"; } else static if(is(T == K[], K)) { auto templ = div.addChild("template"); templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); @@ -9686,30 +10134,34 @@ struct MultipleResponses(T...) { --- auto valueToTest = your_test_function(); - valueToTest.visit!( - (Redirection) { assert(0); }, // got a redirection instead of a string, fail the test + valueToTest.visit( + (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test (string s) { assert(s == "test"); } // right value, go ahead and test it. ); --- + + History: + Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. + It tried to use alias lambdas before, but runtime delegates work much better so I changed it. +/ - void visit(Handlers...)() { - template findHandler(type, HandlersToCheck...) { + void visit(Handlers...)(Handlers handlers) { + template findHandler(type, int count, HandlersToCheck...) { static if(HandlersToCheck.length == 0) - alias findHandler = void; + enum findHandler = -1; else { - static if(is(typeof(HandlersToCheck[0](type.init)))) - alias findHandler = handler; + static if(is(typeof(HandlersToCheck[0].init(type.init)))) + enum findHandler = count; else - alias findHandler = findHandler!(type, HandlersToCheck[1 .. $]); + enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); } } foreach(index, type; T) { - alias handler = findHandler!(type, Handlers); - static if(is(handler == void)) + enum handlerIndex = findHandler!(type, 0, Handlers); + static if(handlerIndex == -1) static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); else { - if(index == contains) - handler(payload[index]); + if(index == this.contains) + handlers[handlerIndex](this.payload[index]); } } } @@ -9795,7 +10247,12 @@ private string nextPieceFromSlash(ref string remainingUrl) { return ident; } +/++ + UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. ++/ enum AddTrailingSlash; +/// ditto +enum RemoveTrailingSlash; private auto serveApiInternal(T)(string urlPrefix) { @@ -9852,7 +10309,7 @@ private auto serveApiInternal(T)(string urlPrefix) { switch(cgi.request("format", "html")) { case "html": static void dummy() {} - presenter.presentExceptionAsHtml!(dummy)(cgi, t, &dummy); + presenter.presentExceptionAsHtml(cgi, t, null); return true; case "json": var envelope = var.emptyObject; @@ -9976,6 +10433,12 @@ private auto serveApiInternal(T)(string urlPrefix) { cgi.setResponseLocation(cgi.pathInfo ~ "/"); return true; } + } else static if(is(attr == RemoveTrailingSlash)) { + if(remainingUrl !is null) { + cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); + return true; + } + } else static if(__traits(isSame, AutomaticForm, attr)) { automaticForm = true; } @@ -10054,47 +10517,25 @@ private auto serveApiInternal(T)(string urlPrefix) { if(callFunction) +/ - if(automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET) { + auto format = cgi.request("format", defaultFormat!overload()); + auto wantsFormFormat = format.startsWith("form-"); + + if(wantsFormFormat || (automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET)) { // Should I still show the form on a json thing? idk... auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); - presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), "html"); + presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), wantsFormFormat ? format["form_".length .. $] : "html"); return true; } - switch(cgi.request("format", defaultFormat!overload())) { - case "html": - // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. - try { - - auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); - presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), "html"); - } catch(Throwable t) { - presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); - } - return true; - case "json": - auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); - static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { - var json; - foreach(index, type; Types) { - if(ret.contains == index) - json = ret.payload[index]; - } - } else { - var json = ret; - } - var envelope = json; // var.emptyObject; - /* - envelope.success = true; - envelope.result = json; - envelope.error = null; - */ - cgi.setResponseContentType("application/json"); - cgi.write(envelope.toJson(), true); - return true; - default: - cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. - return true; + + try { + // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. + auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); + presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); + } catch(Throwable t) { + // presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); + presenter.presentExceptionalReturn(cgi, t, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); } + return true; //}} //cgi.header("Accept: POST"); // FIXME list the real thing @@ -10743,14 +11184,12 @@ bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string u // FIXME: OPTIONS, HEAD } catch(Throwable t) { - presenter.presentExceptionAsHtml!(DUMMY)(cgi, t, null); + presenter.presentExceptionAsHtml(cgi, t); } return true; } -struct DUMMY {} - /+ struct SetOfFields(T) { private void[0][string] storage; @@ -10770,6 +11209,33 @@ enum hideonindex; +/ /++ + Returns true if I recommend gzipping content of this type. You might + want to call it from your Presenter classes before calling cgi.write. + + --- + cgi.setResponseContentType(yourContentType); + cgi.gzipResponse = gzipRecommendedForContentType(yourContentType); + cgi.write(yourData, true); + --- + + This is used internally by [serveStaticFile], [serveStaticFileDirectory], [serveStaticData], and maybe others I forgot to update this doc about. + + + The implementation considers text content to be recommended to gzip. This may change, but it seems reasonable enough for now. + + History: + Added January 28, 2023 (dub v11.0) ++/ +bool gzipRecommendedForContentType(string contentType) { + if(contentType.startsWith("text/")) + return true; + if(contentType.startsWith("application/javascript")) + return true; + + return false; +} + +/++ Serves a static file. To be used with [dispatcher]. See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] @@ -10779,7 +11245,7 @@ auto serveStaticFile(string urlPrefix, string filename = null, string contentTyp // man 2 sendfile assert(urlPrefix[0] == '/'); if(filename is null) - filename = urlPrefix[1 .. $]; + filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? if(contentType is null) { contentType = contentTypeFromFileExtension(filename); } @@ -10790,9 +11256,10 @@ auto serveStaticFile(string urlPrefix, string filename = null, string contentTyp } static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { - if(details.contentType.indexOf("image/") == 0) + if(details.contentType.indexOf("image/") == 0 || details.contentType.indexOf("audio/") == 0) cgi.setCache(true); cgi.setResponseContentType(details.contentType); + cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); cgi.write(std.file.read(details.filename), true); return true; } @@ -10828,6 +11295,8 @@ auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentTyp string contentTypeFromFileExtension(string filename) { if(filename.endsWith(".png")) return "image/png"; + if(filename.endsWith(".apng")) + return "image/apng"; if(filename.endsWith(".svg")) return "image/svg+xml"; if(filename.endsWith(".jpg")) @@ -10842,28 +11311,56 @@ string contentTypeFromFileExtension(string filename) { return "application/wasm"; if(filename.endsWith(".mp3")) return "audio/mpeg"; + if(filename.endsWith(".pdf")) + return "application/pdf"; return null; } /// This serves a directory full of static files, figuring out the content-types from file extensions. /// It does not let you to descend into subdirectories (or ascend out of it, of course) -auto serveStaticFileDirectory(string urlPrefix, string directory = null) { +auto serveStaticFileDirectory(string urlPrefix, string directory = null, bool recursive = false) { assert(urlPrefix[0] == '/'); assert(urlPrefix[$-1] == '/'); static struct DispatcherDetails { string directory; + bool recursive; } if(directory is null) directory = urlPrefix[1 .. $]; + if(directory.length == 0) + directory = "./"; + assert(directory[$-1] == '/'); static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { - auto file = cgi.pathInfo[urlPrefix.length .. $]; - if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) - return false; + auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct + + if(details.recursive) { + // never allow a backslash since it isn't in a typical url anyway and makes the following checks easier + if(file.indexOf("\\") != -1) + return false; + + import std.path; + + file = std.path.buildNormalizedPath(file); + enum upOneDir = ".." ~ std.path.dirSeparator; + + // also no point doing any kind of up directory things since that makes it more likely to break out of the parent + if(file == ".." || file.startsWith(upOneDir)) + return false; + if(std.path.isAbsolute(file)) + return false; + + // FIXME: if it has slashes and stuff, should we redirect to the canonical resource? or what? + + // once it passes these filters it is probably ok. + } else { + if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) + return false; + } auto contentType = contentTypeFromFileExtension(file); @@ -10874,6 +11371,7 @@ auto serveStaticFileDirectory(string urlPrefix, string directory = null) { //else if(contentType.indexOf("audio/") == 0) cgi.setCache(true); cgi.setResponseContentType(contentType); + cgi.gzipResponse = gzipRecommendedForContentType(contentType); cgi.write(std.file.read(fn), true); return true; } else { @@ -10881,7 +11379,7 @@ auto serveStaticFileDirectory(string urlPrefix, string directory = null) { } } - return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory)); + return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory, recursive)); } /++ @@ -10959,14 +11457,162 @@ auto dispatchTo(alias handler)(string urlPrefix) { return DispatcherDefinition!(internalHandler)(urlPrefix, false); } -/+ /++ See [serveStaticFile] if you want to serve a file off disk. + + History: + Added January 28, 2023 (dub v11.0) +/ -auto serveStaticData(string urlPrefix, const(void)[] data, string contentType) { +auto serveStaticData(string urlPrefix, immutable(ubyte)[] data, string contentType, string filenameToSuggestAsDownload = null) { + assert(urlPrefix[0] == '/'); + static struct DispatcherDetails { + immutable(ubyte)[] data; + string contentType; + string filenameToSuggestAsDownload; + } + + static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { + cgi.setCache(true); + cgi.setResponseContentType(details.contentType); + if(details.filenameToSuggestAsDownload.length) + cgi.header("Content-Disposition: attachment; filename=\""~details.filenameToSuggestAsDownload~"\""); + cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); + cgi.write(details.data, true); + return true; + } + return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType, filenameToSuggestAsDownload)); } + +/++ + Placeholder for use with [dispatchSubsection]'s `NewPresenter` argument to indicate you want to keep the parent's presenter. + + History: + Added January 28, 2023 (dub v11.0) +/ +alias KeepExistingPresenter = typeof(null); + +/++ + For use with [dispatchSubsection]. Calls your filter with the request and if your filter returns false, + this issues the given errorCode and stops processing. + + --- + bool hasAdminPermissions(Cgi cgi) { + return true; + } + + mixin DispatcherMain!( + "/admin".dispatchSubsection!( + passFilterOrIssueError!(hasAdminPermissions, 403), + KeepExistingPresenter, + "/".serveApi!AdminFunctions + ) + ); + --- + + History: + Added January 28, 2023 (dub v11.0) ++/ +template passFilterOrIssueError(alias filter, int errorCode) { + bool passFilterOrIssueError(DispatcherDetails)(DispatcherDetails dd) { + if(filter(dd.cgi)) + return true; + dd.presenter.renderBasicError(dd.cgi, errorCode); + return false; + } +} + +/++ + Allows for a subsection of your dispatched urls to be passed through other a pre-request filter, optionally pick up an new presenter class, + and then be dispatched to their own handlers. + + --- + /+ + // a long-form filter function + bool permissionCheck(DispatcherData)(DispatcherData dd) { + // you are permitted to call mutable methods on the Cgi object + // Note if you use a Cgi subclass, you can try dynamic casting it back to your custom type to attach per-request data + // though much of the request is immutable so there's only so much you're allowed to do to modify it. + + if(checkPermissionOnRequest(dd.cgi)) { + return true; // OK, allow processing to continue + } else { + dd.presenter.renderBasicError(dd.cgi, 403); // reply forbidden to the requester + return false; // and stop further processing into this subsection + } + } + +/ + + // but you can also do short-form filters: + + bool permissionCheck(Cgi cgi) { + return ("ok" in cgi.get) !is null; + } + + // handler for the subsection + class AdminClass : WebObject { + int foo() { return 5; } + } + + // handler for the main site + class TheMainSite : WebObject {} + + mixin DispatcherMain!( + "/admin".dispatchSubsection!( + // converts our short-form filter into a long-form filter + passFilterOrIssueError!(permissionCheck, 403), + // can use a new presenter if wanted for the subsection + KeepExistingPresenter, + // and then provide child route dispatchers + "/".serveApi!AdminClass + ), + // and back to the top level + "/".serveApi!TheMainSite + ); + --- + + Note you can encapsulate sections in files like this: + + --- + auto adminDispatcher(string urlPrefix) { + return urlPrefix.dispatchSubsection!( + .... + ); + } + + mixin DispatcherMain!( + "/admin".adminDispatcher, + // and so on + ) + --- + + If you want no filter, you can pass `(cgi) => true` as the filter to approve all requests. + + If you want to keep the same presenter as the parent, use [KeepExistingPresenter] as the presenter argument. + + + History: + Added January 28, 2023 (dub v11.0) ++/ +auto dispatchSubsection(alias PreRequestFilter, NewPresenter, definitions...)(string urlPrefix) { + assert(urlPrefix[0] == '/'); + assert(urlPrefix[$-1] != '/'); + static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { + static if(!is(PreRequestFilter == typeof(null))) { + if(!PreRequestFilter(DispatcherData!Presenter(cgi, presenter, urlPrefix.length))) + return true; // we handled it by rejecting it + } + + static if(is(NewPresenter == Presenter) || is(NewPresenter == typeof(null))) { + return dispatcher!definitions(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); + } else { + auto newPresenter = new NewPresenter(); + return dispatcher!(definitions(DispatcherData!NewPresenter(cgi, presenter, urlPrefix.length))); + } + } + + return DispatcherDefinition!(internalHandler)(urlPrefix, false); +} /++ A URL dispatcher. @@ -10983,10 +11629,11 @@ auto serveStaticData(string urlPrefix, const(void)[] data, string contentType) { You define a series of url prefixes followed by handlers. - [dispatchTo] will send the request to another function for handling. You may want to do different pre- and post- processing there, for example, an authorization check and different page layout. You can use different - presenters and different function chains. NOT IMPLEMENTED + presenters and different function chains. See [dispatchSubsection] for details. + + [dispatchTo] will send the request to another function for handling. +/ template dispatcher(definitions...) { bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { @@ -11170,11 +11817,11 @@ bool apiDispatcher()(Cgi cgi) { version(linux) private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; /* -Copyright: Adam D. Ruppe, 2008 - 2021 +Copyright: Adam D. Ruppe, 2008 - 2023 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. Authors: Adam D. Ruppe - Copyright Adam D. Ruppe 2008 - 2021. + Copyright Adam D. Ruppe 2008 - 2023. Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) diff --git a/src/ext_depends/arsd/core.d b/src/ext_depends/arsd/core.d new file mode 100644 index 0000000..23a699d --- /dev/null +++ b/src/ext_depends/arsd/core.d @@ -0,0 +1,6106 @@ +/++ + Please note: the api and behavior of this module is not externally stable at this time. See the documentation on specific functions. + + Shared core functionality including exception helpers, library loader, event loop, and possibly more. Maybe command line processor and uda helper and some basic shared annotation types. + + I'll probably move the url, websocket, and ssl stuff in here too as they are often shared. Maybe a small internationalization helper type (a hook for external implementation) and COM helpers too. + + If you use this directly outside the arsd library, you might consider using `static import` since names in here are likely to clash with Phobos if you use them together. `static import` will let you easily disambiguate and avoid name conflict errors if I add more here. Some names even clash deliberately to remind me to avoid some antipatterns inside the arsd modules! + + History: + Added March 2023 (dub v11.0). Several functions were migrated in here at that time, noted individually. Members without a note were added with the module. ++/ +module arsd.core; + +// FIXME: add callbacks on file open for tracing dependencies dynamically + +// see for useful info: https://devblogs.microsoft.com/dotnet/how-async-await-really-works/ + +// see: https://wiki.openssl.org/index.php/Simple_TLS_Server + +import core.thread; +import core.volatile; +import core.atomic; +import core.time; + +import core.stdc.errno; + +import core.attribute; +static if(!__traits(hasMember, core.attribute, "mustuse")) + enum mustuse; + +// FIXME: add an arena allocator? can do task local destruction maybe. + +// the three implementations are windows, epoll, and kqueue +version(Windows) { + version=Arsd_core_windows; + + // import core.sys.windows.windows; + import core.sys.windows.winbase; + import core.sys.windows.windef; + import core.sys.windows.winnls; + import core.sys.windows.winuser; + import core.sys.windows.winsock2; + + pragma(lib, "user32"); + pragma(lib, "ws2_32"); +} else version(linux) { + version=Arsd_core_epoll; + + version=Arsd_core_has_cloexec; +} else version(FreeBSD) { + version=Arsd_core_kqueue; + + import core.sys.freebsd.sys.event; +} else version(DragonFlyBSD) { + // NOT ACTUALLY TESTED + version=Arsd_core_kqueue; + + import core.sys.dragonflybsd.sys.event; +} else version(NetBSD) { + // NOT ACTUALLY TESTED + version=Arsd_core_kqueue; + + import core.sys.netbsd.sys.event; +} else version(OpenBSD) { + version=Arsd_core_kqueue; + + // THIS FILE DOESN'T ACTUALLY EXIST, WE NEED TO MAKE IT + import core.sys.openbsd.sys.event; +} else version(OSX) { + version=Arsd_core_kqueue; + + import core.sys.darwin.sys.event; +} + +version(Posix) { + import core.sys.posix.signal; + import core.sys.posix.unistd; + + import core.sys.posix.sys.un; + import core.sys.posix.sys.socket; + import core.sys.posix.netinet.in_; +} + +// FIXME: the exceptions should actually give some explanatory text too (at least sometimes) + +/+ + ========================= + GENERAL UTILITY FUNCTIONS + ========================= ++/ + +// enum stringz : const(char)* { init = null } + +/++ + A wrapper around a `const(char)*` to indicate that it is a zero-terminated C string. ++/ +struct stringz { + private const(char)* raw; + + /++ + Wraps the given pointer in the struct. Note that it retains a copy of the pointer. + +/ + this(const(char)* raw) { + this.raw = raw; + } + + /++ + Returns the original raw pointer back out. + +/ + const(char)* ptr() const { + return raw; + } + + /++ + Borrows a slice of the pointer up to (but not including) the zero terminator. + +/ + const(char)[] borrow() const { + if(raw is null) + return null; + + const(char)* p = raw; + int length; + while(*p++) length++; + + return raw[0 .. length]; + } +} + +/++ + A limited variant to hold just a few types. It is made for the use of packing a small amount of extra data into error messages. ++/ +/+ + * if length and ptr are both 0, it is null + * if ptr == 1, length is an integer + * if ptr == 2, length is an unsigned integer (suggest printing in hex) + * if ptr == 3, length is a combination of flags (suggest printing in binary) + * if ptr == 4, length is a unix permission thing (suggest printing in octal) + * if ptr == 5, length is a double float + * if ptr == 15, length must be 0. this holds an empty, non-null, SSO string. + * if ptr >= 16 && < 24, length is reinterpret-casted a small string of length of (ptr & 0x7) + 1 + * if length == size_t.max, ptr is interpreted as a stringz + * if ptr >= 1024, it is a non-null D string or byte array. It is a string if the length high bit is clear, a byte array if it is set. the length is what is left after you mask that out. + + All other ptr values are reserved for future expansion. ++/ +struct LimitedVariant { + + /++ + + +/ + enum Contains { + null_, + intDecimal, + intHex, + intBinary, + intOctal, + double_, + emptySso, + stringSso, + stringz, + string, + bytes, + + invalid, + } + + /++ + + +/ + Contains contains() const { + auto tag = cast(size_t) ptr; + if(ptr is null && length is null) + return Contains.null_; + else switch(tag) { + case 1: return Contains.intDecimal; + case 2: return Contains.intHex; + case 3: return Contains.intBinary; + case 4: return Contains.intOctal; + case 5: return Contains.double_; + case 15: return length is null ? Contains.emptySso : Contains.invalid; + default: + if(tag >= 16 && tag < 24) { + return Contains.stringSso; + } else if(tag >= 1024) { + if(cast(size_t) length == size_t.max) + return Contains.stringz; + else + return isHighBitSet ? Contains.bytes : Contains.string; + } else { + return Contains.invalid; + } + } + } + + /// ditto + bool containsInt() const { + with(Contains) + switch(contains) { + case intDecimal, intHex, intBinary, intOctal: + return true; + default: + return false; + } + } + + /// ditto + bool containsString() const { + with(Contains) + switch(contains) { + case null_, emptySso, stringSso, string: + // case stringz: + return true; + default: + return false; + } + } + + /// ditto + bool containsDouble() const { + with(Contains) + switch(contains) { + case double_: + return true; + default: + return false; + } + } + + /// ditto + bool containsBytes() const { + with(Contains) + switch(contains) { + case bytes, null_: + return true; + default: + return false; + } + } + + private const(void)* length; + private const(ubyte)* ptr; + + private void Throw() const { + throw ArsdException!"LimitedVariant"(cast(size_t) length, cast(size_t) ptr); + } + + private bool isHighBitSet() const { + return (cast(size_t) length >> (size_t.sizeof * 8 - 1) & 0x1) != 0; + } + + /++ + getString gets a reference to the string stored internally, see [toString] to get a string representation or whatever is inside. + + +/ + const(char)[] getString() const return { + with(Contains) + switch(contains()) { + case null_: + return null; + case emptySso: + return (cast(const(char)*) ptr)[0 .. 0]; // zero length, non-null + case stringSso: + auto len = ((cast(size_t) ptr) & 0x7) + 1; + return (cast(char*) &length)[0 .. len]; + case string: + return (cast(const(char)*) ptr)[0 .. cast(size_t) length]; + default: + Throw(); assert(0); + } + } + + /// ditto + long getInt() const { + if(containsInt) + return cast(long) length; + else + Throw(); + assert(0); + } + + /// ditto + double getDouble() const { + if(containsDouble) + return *cast(double*) &length; + else + Throw(); + assert(0); + } + + /// ditto + const(ubyte)[] getBytes() const { + with(Contains) + switch(contains()) { + case null_: + return null; + case bytes: + return ptr[0 .. (cast(size_t) length) & ((1UL << (size_t.sizeof * 8 - 1)) - 1)]; + default: + Throw(); assert(0); + } + } + + /++ + + +/ + string toString() const { + + string intHelper(string prefix, int radix) { + char[128] buffer; + buffer[0 .. prefix.length] = prefix[]; + char[] toUse = buffer[prefix.length .. $]; + + auto got = intToString(getInt(), toUse[], IntToStringArgs().withRadix(radix)); + + return buffer[0 .. prefix.length + got.length].idup; + } + + with(Contains) + final switch(contains()) { + case null_: + return "<null>"; + case intDecimal: + return intHelper("", 10); + case intHex: + return intHelper("0x", 16); + case intBinary: + return intHelper("0b", 2); + case intOctal: + return intHelper("0o", 8); + case emptySso, stringSso, string: + return getString().idup; + case bytes: + auto b = getBytes(); + + return "<bytes>"; // FIXME + + case double_: + assert(0); // FIXME + case stringz: + assert(0); // FIXME + case invalid: + return "<invalid>"; + } + } + + /++ + + +/ + this(string s) { + ptr = cast(const(ubyte)*) s.ptr; + length = cast(void*) s.length; + } + + /// ditto + this(const(ubyte)[] b) { + ptr = cast(const(ubyte)*) b.ptr; + length = cast(void*) (b.length | (1UL << (size_t.sizeof * 8 - 1))); + } + + /// ditto + this(long l, int base = 10) { + int tag; + switch(base) { + case 10: tag = 1; break; + case 16: tag = 2; break; + case 2: tag = 3; break; + case 8: tag = 4; break; + default: assert(0, "You passed an invalid base to LimitedVariant"); + } + ptr = cast(ubyte*) tag; + length = cast(void*) l; + } + + /// ditto + version(none) + this(double d) { + // this crashes dmd! omg + assert(0); + // ptr = cast(ubyte*) 15; + // length = cast(void*) *cast(size_t*) &d; + } +} + +unittest { + LimitedVariant v = LimitedVariant("foo"); + assert(v.containsString()); + assert(!v.containsInt()); + assert(v.getString() == "foo"); + + LimitedVariant v2 = LimitedVariant(4); + assert(v2.containsInt()); + assert(!v2.containsString()); + assert(v2.getInt() == 4); + + LimitedVariant v3 = LimitedVariant(cast(ubyte[]) [1, 2, 3]); + assert(v3.containsBytes()); + assert(!v3.containsString()); + assert(v3.getBytes() == [1, 2, 3]); +} + +/++ + This is a dummy type to indicate the end of normal arguments and the beginning of the file/line inferred args. It is meant to ensure you don't accidentally send a string that is interpreted as a filename when it was meant to be a normal argument to the function and trigger the wrong overload. ++/ +struct ArgSentinel {} + +/++ + A trivial wrapper around C's malloc that creates a D slice. It multiples n by T.sizeof and returns the slice of the pointer from 0 to n. + + Please note that the ptr might be null - it is your responsibility to check that, same as normal malloc. Check `ret is null` specifically, since `ret.length` will always be `n`, even if the `malloc` failed. + + Remember to `free` the returned pointer with `core.stdc.stdlib.free(ret.ptr);` + + $(TIP + I strongly recommend you simply use the normal garbage collector unless you have a very specific reason not to. + ) + + See_Also: + [mallocedStringz] ++/ +T[] mallocSlice(T)(size_t n) { + import c = core.stdc.stdlib; + + return (cast(T*) c.malloc(n * T.sizeof))[0 .. n]; +} + +/++ + Uses C's malloc to allocate a copy of `original` with an attached zero terminator. It may return a slice with a `null` pointer (but non-zero length!) if `malloc` fails and you are responsible for freeing the returned pointer with `core.stdc.stdlib.free(ret.ptr)`. + + $(TIP + I strongly recommend you use [CharzBuffer] or Phobos' [std.string.toStringz] instead unless there's a special reason not to. + ) + + See_Also: + [CharzBuffer] for a generally better alternative. You should only use `mallocedStringz` where `CharzBuffer` cannot be used (e.g. when druntime is not usable or you have no stack space for the temporary buffer). + + [mallocSlice] is the function this function calls, so the notes in its documentation applies here too. ++/ +char[] mallocedStringz(in char[] original) { + auto slice = mallocSlice!char(original.length + 1); + if(slice is null) + return null; + slice[0 .. original.length] = original[]; + slice[original.length] = 0; + return slice; +} + +/++ + Basically a `scope class` you can return from a function or embed in another aggregate. ++/ +struct OwnedClass(Class) { + ubyte[__traits(classInstanceSize, Class)] rawData; + + static OwnedClass!Class defaultConstructed() { + OwnedClass!Class i = OwnedClass!Class.init; + i.initializeRawData(); + return i; + } + + private void initializeRawData() @trusted { + if(!this) + rawData[] = cast(ubyte[]) typeid(Class).initializer[]; + } + + this(T...)(T t) { + initializeRawData(); + rawInstance.__ctor(t); + } + + bool opCast(T : bool)() @trusted { + return !(*(cast(void**) rawData.ptr) is null); + } + + @disable this(); + @disable this(this); + + Class rawInstance() return @trusted { + if(!this) + throw new Exception("null"); + return cast(Class) rawData.ptr; + } + + alias rawInstance this; + + ~this() @trusted { + if(this) + .destroy(rawInstance()); + } +} + + + +version(Posix) +package(arsd) void makeNonBlocking(int fd) { + import core.sys.posix.fcntl; + auto flags = fcntl(fd, F_GETFL, 0); + if(flags == -1) + throw new ErrnoApiException("fcntl get", errno); + flags |= O_NONBLOCK; + auto s = fcntl(fd, F_SETFL, flags); + if(s == -1) + throw new ErrnoApiException("fcntl set", errno); +} + +version(Posix) +package(arsd) void setCloExec(int fd) { + import core.sys.posix.fcntl; + auto flags = fcntl(fd, F_GETFD, 0); + if(flags == -1) + throw new ErrnoApiException("fcntl get", errno); + flags |= FD_CLOEXEC; + auto s = fcntl(fd, F_SETFD, flags); + if(s == -1) + throw new ErrnoApiException("fcntl set", errno); +} + + +/++ + A helper object for temporarily constructing a string appropriate for the Windows API from a D UTF-8 string. + + + It will use a small internal static buffer is possible, and allocate a new buffer if the string is too big. + + History: + Moved from simpledisplay.d to core.d in March 2023 (dub v11.0). ++/ +version(Windows) +struct WCharzBuffer { + private wchar[] buffer; + private wchar[128] staticBuffer = void; + + /// Length of the string, excluding the zero terminator. + size_t length() { + return buffer.length; + } + + // Returns the pointer to the internal buffer. You must assume its lifetime is less than that of the WCharzBuffer. It is zero-terminated. + wchar* ptr() { + return buffer.ptr; + } + + /// Returns the slice of the internal buffer, excluding the zero terminator (though there is one present right off the end of the slice). You must assume its lifetime is less than that of the WCharzBuffer. + wchar[] slice() { + return buffer; + } + + /// Copies it into a static array of wchars + void copyInto(R)(ref R r) { + static if(is(R == wchar[N], size_t N)) { + r[0 .. this.length] = slice[]; + r[this.length] = 0; + } else static assert(0, "can only copy into wchar[n], not " ~ R.stringof); + } + + /++ + conversionFlags = [WindowsStringConversionFlags] + +/ + this(in char[] data, int conversionFlags = 0) { + conversionFlags |= WindowsStringConversionFlags.zeroTerminate; // this ALWAYS zero terminates cuz of its name + auto sz = sizeOfConvertedWstring(data, conversionFlags); + if(sz > staticBuffer.length) + buffer = new wchar[](sz); + else + buffer = staticBuffer[]; + + buffer = makeWindowsString(data, buffer, conversionFlags); + } +} + +/++ + Alternative for toStringz + + History: + Added March 18, 2023 (dub v11.0) ++/ +struct CharzBuffer { + private char[] buffer; + private char[128] staticBuffer = void; + + /// Length of the string, excluding the zero terminator. + size_t length() { + assert(buffer.length > 0); + return buffer.length - 1; + } + + // Returns the pointer to the internal buffer. You must assume its lifetime is less than that of the CharzBuffer. It is zero-terminated. + char* ptr() { + return buffer.ptr; + } + + /// Returns the slice of the internal buffer, excluding the zero terminator (though there is one present right off the end of the slice). You must assume its lifetime is less than that of the CharzBuffer. + char[] slice() { + assert(buffer.length > 0); + return buffer[0 .. $-1]; + } + + /// Copies it into a static array of chars + void copyInto(R)(ref R r) { + static if(is(R == char[N], size_t N)) { + r[0 .. this.length] = slice[]; + r[this.length] = 0; + } else static assert(0, "can only copy into char[n], not " ~ R.stringof); + } + + @disable this(); + @disable this(this); + + /++ + Copies `data` into the CharzBuffer, allocating a new one if needed, and zero-terminates it. + +/ + this(in char[] data) { + if(data.length + 1 > staticBuffer.length) + buffer = new char[](data.length + 1); + else + buffer = staticBuffer[]; + + buffer[0 .. data.length] = data[]; + buffer[data.length] = 0; + } +} + +/++ + Given the string `str`, converts it to a string compatible with the Windows API and puts the result in `buffer`, returning the slice of `buffer` actually used. `buffer` must be at least [sizeOfConvertedWstring] elements long. + + History: + Moved from simpledisplay.d to core.d in March 2023 (dub v11.0). ++/ +version(Windows) +wchar[] makeWindowsString(in char[] str, wchar[] buffer, int conversionFlags = WindowsStringConversionFlags.zeroTerminate) { + if(str.length == 0) + return null; + + int pos = 0; + dchar last; + foreach(dchar c; str) { + if(c <= 0xFFFF) { + if((conversionFlags & WindowsStringConversionFlags.convertNewLines) && c == 10 && last != 13) + buffer[pos++] = 13; + buffer[pos++] = cast(wchar) c; + } else if(c <= 0x10FFFF) { + buffer[pos++] = cast(wchar)((((c - 0x10000) >> 10) & 0x3FF) + 0xD800); + buffer[pos++] = cast(wchar)(((c - 0x10000) & 0x3FF) + 0xDC00); + } + + last = c; + } + + if(conversionFlags & WindowsStringConversionFlags.zeroTerminate) { + buffer[pos] = 0; + } + + return buffer[0 .. pos]; +} + +/++ + Converts the Windows API string `str` to a D UTF-8 string, storing it in `buffer`. Returns the slice of `buffer` actually used. + + History: + Moved from simpledisplay.d to core.d in March 2023 (dub v11.0). ++/ +version(Windows) +char[] makeUtf8StringFromWindowsString(in wchar[] str, char[] buffer) { + if(str.length == 0) + return null; + + auto got = WideCharToMultiByte(CP_UTF8, 0, str.ptr, cast(int) str.length, buffer.ptr, cast(int) buffer.length, null, null); + if(got == 0) { + if(GetLastError() == ERROR_INSUFFICIENT_BUFFER) + throw new object.Exception("not enough buffer"); + else + throw new object.Exception("conversion"); // FIXME: GetLastError + } + return buffer[0 .. got]; +} + +/++ + Converts the Windows API string `str` to a newly-allocated D UTF-8 string. + + History: + Moved from simpledisplay.d to core.d in March 2023 (dub v11.0). ++/ +version(Windows) +string makeUtf8StringFromWindowsString(in wchar[] str) { + char[] buffer; + auto got = WideCharToMultiByte(CP_UTF8, 0, str.ptr, cast(int) str.length, null, 0, null, null); + buffer.length = got; + + // it is unique because we just allocated it above! + return cast(string) makeUtf8StringFromWindowsString(str, buffer); +} + +/// ditto +version(Windows) +string makeUtf8StringFromWindowsString(wchar* str) { + char[] buffer; + auto got = WideCharToMultiByte(CP_UTF8, 0, str, -1, null, 0, null, null); + buffer.length = got; + + got = WideCharToMultiByte(CP_UTF8, 0, str, -1, buffer.ptr, cast(int) buffer.length, null, null); + if(got == 0) { + if(GetLastError() == ERROR_INSUFFICIENT_BUFFER) + throw new object.Exception("not enough buffer"); + else + throw new object.Exception("conversion"); // FIXME: GetLastError + } + return cast(string) buffer[0 .. got]; +} + +// only used from minigui rn +package int findIndexOfZero(in wchar[] str) { + foreach(idx, wchar ch; str) + if(ch == 0) + return cast(int) idx; + return cast(int) str.length; +} +package int findIndexOfZero(in char[] str) { + foreach(idx, char ch; str) + if(ch == 0) + return cast(int) idx; + return cast(int) str.length; +} + +/++ + Returns a minimum buffer length to hold the string `s` with the given conversions. It might be slightly larger than necessary, but is guaranteed to be big enough to hold it. + + History: + Moved from simpledisplay.d to core.d in March 2023 (dub v11.0). ++/ +version(Windows) +int sizeOfConvertedWstring(in char[] s, int conversionFlags) { + int size = 0; + + if(conversionFlags & WindowsStringConversionFlags.convertNewLines) { + // need to convert line endings, which means the length will get bigger. + + // BTW I betcha this could be faster with some simd stuff. + char last; + foreach(char ch; s) { + if(ch == 10 && last != 13) + size++; // will add a 13 before it... + size++; + last = ch; + } + } else { + // no conversion necessary, just estimate based on length + /* + I don't think there's any string with a longer length + in code units when encoded in UTF-16 than it has in UTF-8. + This will probably over allocate, but that's OK. + */ + size = cast(int) s.length; + } + + if(conversionFlags & WindowsStringConversionFlags.zeroTerminate) + size++; + + return size; +} + +/++ + Used by [makeWindowsString] and [WCharzBuffer] + + History: + Moved from simpledisplay.d to core.d in March 2023 (dub v11.0). ++/ +version(Windows) +enum WindowsStringConversionFlags : int { + /++ + Append a zero terminator to the string. + +/ + zeroTerminate = 1, + /++ + Converts newlines from \n to \r\n. + +/ + convertNewLines = 2, +} + +/++ + An int printing function that doesn't need to import Phobos. Can do some of the things std.conv.to and std.format.format do. + + The buffer must be sized to hold the converted number. 32 chars is enough for most anything. + + Returns: the slice of `buffer` containing the converted number. ++/ +char[] intToString(long value, char[] buffer, IntToStringArgs args = IntToStringArgs.init) { + const int radix = args.radix ? args.radix : 10; + const int digitsPad = args.padTo; + const int groupSize = args.groupSize; + + int pos; + + if(value < 0) { + buffer[pos++] = '-'; + value = -value; + } + + int start = pos; + int digitCount; + + do { + auto remainder = value % radix; + value = value / radix; + + buffer[pos++] = cast(char) (remainder < 10 ? (remainder + '0') : (remainder - 10 + args.ten)); + digitCount++; + } while(value); + + if(digitsPad > 0) { + while(digitCount < digitsPad) { + buffer[pos++] = args.padWith; + digitCount++; + } + } + + assert(pos >= 1); + assert(pos - start > 0); + + auto reverseSlice = buffer[start .. pos]; + for(int i = 0; i < reverseSlice.length / 2; i++) { + auto paired = cast(int) reverseSlice.length - i - 1; + char tmp = reverseSlice[i]; + reverseSlice[i] = reverseSlice[paired]; + reverseSlice[paired] = tmp; + } + + return buffer[0 .. pos]; +} + +/// ditto +struct IntToStringArgs { + private { + ubyte padTo; + char padWith; + ubyte radix; + char ten; + ubyte groupSize; + char separator; + } + + IntToStringArgs withPadding(int padTo, char padWith = '0') { + IntToStringArgs args = this; + args.padTo = cast(ubyte) padTo; + args.padWith = padWith; + return args; + } + + IntToStringArgs withRadix(int radix, char ten = 'a') { + IntToStringArgs args = this; + args.radix = cast(ubyte) radix; + args.ten = ten; + return args; + } + + IntToStringArgs withGroupSeparator(int groupSize, char separator = '_') { + IntToStringArgs args = this; + args.groupSize = cast(ubyte) groupSize; + args.separator = separator; + return args; + } +} + +unittest { + char[32] buffer; + assert(intToString(0, buffer[]) == "0"); + assert(intToString(-1, buffer[]) == "-1"); + assert(intToString(-132, buffer[]) == "-132"); + assert(intToString(-1932, buffer[]) == "-1932"); + assert(intToString(1, buffer[]) == "1"); + assert(intToString(132, buffer[]) == "132"); + assert(intToString(1932, buffer[]) == "1932"); + + assert(intToString(0x1, buffer[], IntToStringArgs().withRadix(16)) == "1"); + assert(intToString(0x1b, buffer[], IntToStringArgs().withRadix(16)) == "1b"); + assert(intToString(0xef1, buffer[], IntToStringArgs().withRadix(16)) == "ef1"); + + assert(intToString(0xef1, buffer[], IntToStringArgs().withRadix(16).withPadding(8)) == "00000ef1"); + assert(intToString(-0xef1, buffer[], IntToStringArgs().withRadix(16).withPadding(8)) == "-00000ef1"); + assert(intToString(-0xef1, buffer[], IntToStringArgs().withRadix(16, 'A').withPadding(8, ' ')) == "- EF1"); +} + +/++ + History: + Moved from color.d to core.d in March 2023 (dub v11.0). ++/ +nothrow @safe @nogc pure +inout(char)[] stripInternal(return inout(char)[] s) { + foreach(i, char c; s) + if(c != ' ' && c != '\t' && c != '\n' && c != '\r') { + s = s[i .. $]; + break; + } + for(int a = cast(int)(s.length - 1); a > 0; a--) { + char c = s[a]; + if(c != ' ' && c != '\t' && c != '\n' && c != '\r') { + s = s[0 .. a + 1]; + break; + } + } + + return s; +} + +nothrow @safe @nogc pure +inout(char)[] stripRightInternal(return inout(char)[] s) { + for(int a = cast(int)(s.length - 1); a > 0; a--) { + char c = s[a]; + if(c != ' ' && c != '\t' && c != '\n' && c != '\r') { + s = s[0 .. a + 1]; + break; + } + } + + return s; + +} + +/++ + Shortcut for converting some types to string without invoking Phobos (but it will as a last resort). + + History: + Moved from color.d to core.d in March 2023 (dub v11.0). ++/ +string toStringInternal(T)(T t) { + char[32] buffer; + static if(is(T : string)) + return t; + else static if(is(T : long)) + return intToString(t, buffer[]).idup; + else static if(is(T == enum)) { + switch(t) { + foreach(memberName; __traits(allMembers, T)) { + case __traits(getMember, T, memberName): + return memberName; + } + default: + return "<unknown>"; + } + } else { + import std.conv; + return to!string(t); + } +} + +/++ + ++/ +string flagsToString(Flags)(ulong value) { + string r; + + void add(string memberName) { + if(r.length) + r ~= " | "; + r ~= memberName; + } + + string none = "<none>"; + + foreach(memberName; __traits(allMembers, Flags)) { + auto flag = cast(ulong) __traits(getMember, Flags, memberName); + if(flag) { + if((value & flag) == flag) + add(memberName); + } else { + none = memberName; + } + } + + if(r.length == 0) + r = none; + + return r; +} + +unittest { + enum MyFlags { + none = 0, + a = 1, + b = 2 + } + + assert(flagsToString!MyFlags(3) == "a | b"); + assert(flagsToString!MyFlags(0) == "none"); + assert(flagsToString!MyFlags(2) == "b"); +} + +/++ + This populates a struct from a list of values (or other expressions, but it only looks at the values) based on types of the members, with one exception: `bool` members.. maybe. + + It is intended for collecting a record of relevant UDAs off a symbol in a single call like this: + + --- + struct Name { + string n; + } + + struct Validator { + string regex; + } + + struct FormInfo { + Name name; + Validator validator; + } + + @Name("foo") @Validator(".*") + void foo() {} + + auto info = populateFromUdas!(FormInfo, __traits(getAttributes, foo)); + assert(info.name == Name("foo")); + assert(info.validator == Validator(".*")); + --- + + Note that instead of UDAs, you can also pass a variadic argument list and get the same result, but the function is `populateFromArgs` and you pass them as the runtime list to bypass "args cannot be evaluated at compile time" errors: + + --- + void foo(T...)(T t) { + auto info = populateFromArgs!(FormInfo)(t); + // assuming the call below + assert(info.name == Name("foo")); + assert(info.validator == Validator(".*")); + } + + foo(Name("foo"), Validator(".*")); + --- + + The benefit of this over constructing the struct directly is that the arguments can be reordered or missing. Its value is diminished with named arguments in the language. ++/ +template populateFromUdas(Struct, UDAs...) { + enum Struct populateFromUdas = () { + Struct ret; + foreach(memberName; __traits(allMembers, Struct)) { + alias memberType = typeof(__traits(getMember, Struct, memberName)); + foreach(uda; UDAs) { + static if(is(memberType == PresenceOf!a, a)) { + static if(__traits(isSame, a, uda)) + __traits(getMember, ret, memberName) = true; + } + else + static if(is(typeof(uda) : memberType)) { + __traits(getMember, ret, memberName) = uda; + } + } + } + + return ret; + }(); +} + +/// ditto +Struct populateFromArgs(Struct, Args...)(Args args) { + Struct ret; + foreach(memberName; __traits(allMembers, Struct)) { + alias memberType = typeof(__traits(getMember, Struct, memberName)); + foreach(arg; args) { + static if(is(typeof(arg == memberType))) { + __traits(getMember, ret, memberName) = arg; + } + } + } + + return ret; +} + +/// ditto +struct PresenceOf(alias a) { + bool there; + alias there this; +} + +/// +unittest { + enum a; + enum b; + struct Name { string name; } + struct Info { + Name n; + PresenceOf!a athere; + PresenceOf!b bthere; + int c; + } + + void test() @a @Name("test") {} + + auto info = populateFromUdas!(Info, __traits(getAttributes, test)); + assert(info.n == Name("test")); // but present ones are in there + assert(info.athere == true); // non-values can be tested with PresenceOf!it, which works like a bool + assert(info.bthere == false); + assert(info.c == 0); // absent thing will keep the default value +} + +/++ + Declares a delegate property with several setters to allow for handlers that don't care about the arguments. + + Throughout the arsd library, you will often see types of these to indicate that you can set listeners with or without arguments. If you care about the details of the callback event, you can set a delegate that declares them. And if you don't, you can set one that doesn't even declare them and it will be ignored. ++/ +struct FlexibleDelegate(DelegateType) { + // please note that Parameters and ReturnType are public now! + static if(is(DelegateType FunctionType == delegate)) + static if(is(FunctionType Parameters == __parameters)) + static if(is(DelegateType ReturnType == return)) { + + /++ + Calls the currently set delegate. + + Diagnostics: + If the callback delegate has not been set, this may cause a null pointer dereference. + +/ + ReturnType opCall(Parameters args) { + return dg(args); + } + + /++ + Use `if(thing)` to check if the delegate is null or not. + +/ + bool opCast(T : bool)() { + return dg !is null; + } + + /++ + These opAssign overloads are what puts the flexibility in the flexible delegate. + + Bugs: + The other overloads do not keep attributes like `nothrow` on the `dg` parameter, making them unusable if `DelegateType` requires them. I consider the attributes more trouble than they're worth anyway, and the language's poor support for composing them doesn't help any. I have no need for them and thus no plans to add them in the overloads at this time. + +/ + void opAssign(DelegateType dg) { + this.dg = dg; + } + + /// ditto + void opAssign(ReturnType delegate() dg) { + this.dg = (Parameters ignored) => dg(); + } + + /// ditto + void opAssign(ReturnType function(Parameters params) dg) { + this.dg = (Parameters params) => dg(params); + } + + /// ditto + void opAssign(ReturnType function() dg) { + this.dg = (Parameters ignored) => dg(); + } + + /// ditto + void opAssign(typeof(null) explicitNull) { + this.dg = null; + } + + private DelegateType dg; + } + else static assert(0, DelegateType.stringof ~ " failed return value check"); + else static assert(0, DelegateType.stringof ~ " failed parameters check"); + else static assert(0, DelegateType.stringof ~ " failed delegate check"); +} + +/++ + ++/ +unittest { + // you don't have to put the arguments in a struct, but i recommend + // you do as it is more future proof - you can add more info to the + // struct without breaking user code that consumes it. + struct MyEventArguments { + + } + + // then you declare it just adding FlexibleDelegate!() around the + // plain delegate type you'd normally use + FlexibleDelegate!(void delegate(MyEventArguments args)) callback; + + // until you set it, it will be null and thus be false in any boolean check + assert(!callback); + + // can set it to the properly typed thing + callback = delegate(MyEventArguments args) {}; + + // and now it is no longer null + assert(callback); + + // or if you don't care about the args, you can leave them off + callback = () {}; + + // and it works if the compiler types you as a function instead of delegate too + // (which happens automatically if you don't access any local state or if you + // explicitly define it as a function) + + callback = function(MyEventArguments args) { }; + + // can set it back to null explicitly if you ever wanted + callback = null; + + // the reflection info used internally also happens to be exposed publicly + // which can actually sometimes be nice so if the language changes, i'll change + // the code to keep this working. + static assert(is(callback.ReturnType == void)); + + // which can be convenient if the params is an annoying type since you can + // consistently use something like this too + callback = (callback.Parameters params) {}; + + // check for null and call it pretty normally + if(callback) + callback(MyEventArguments()); +} + +/+ + ====================== + ERROR HANDLING HELPERS + ====================== ++/ + +/+ + + arsd code shouldn't be using Exception. Really, I don't think any code should be - instead, construct an appropriate object with structured information. + + If you want to catch someone else's Exception, use `catch(object.Exception e)`. ++/ +//package deprecated struct Exception {} + + +/++ + Base class representing my exceptions. You should almost never work with this directly, but you might catch it as a generic thing. Catch it before generic `object.Exception` or `object.Throwable` in any catch chains. + + + $(H3 General guidelines for exceptions) + + The purpose of an exception is to cancel a task that has proven to be impossible and give the programmer enough information to use at a higher level to decide what to do about it. + + Cancelling a task is accomplished with the `throw` keyword. The transmission of information to a higher level is done by the language runtime. The decision point is marked by the `catch` keyword. The part missing - the job of the `Exception` class you construct and throw - is to gather the information that will be useful at a later decision point. + + It is thus important that you gather as much useful information as possible and keep it in a way that the code catching the exception can still interpret it when constructing an exception. Other concerns are secondary to this to this primary goal. + + With this in mind, here's some guidelines for exception handling in arsd code. + + $(H4 Allocations and lifetimes) + + Don't get clever with exception allocations. You don't know what the catcher is going to do with an exception and you don't want the error handling scheme to introduce its own tricky bugs. Remember, an exception object's first job is to deliver useful information up the call chain in a way this code can use it. You don't know what this code is or what it is going to do. + + Keep your memory management schemes simple and let the garbage collector do its job. + + $(LIST + * All thrown exceptions should be allocated with the `new` keyword. + + * Members inside the exception should be value types or have infinite lifetime (that is, be GC managed). + + * While this document is concerned with throwing, you might want to add additional information to an in-flight exception, and this is done by catching, so you need to know how that works too, and there is a global compiler switch that can change things, so even inside arsd we can't completely avoid its implications. + + DIP1008's presence complicates things a bit on the catch side - if you catch an exception and return it from a function, remember to `ex.refcount = ex.refcount + 1;` so you don't introduce more use-after-free woes for those unfortunate souls. + ) + + $(H4 Error strings) + + Strings can deliver useful information to people reading the message, but are often suboptimal for delivering useful information to other chunks of code. Remember, an exception's first job is to be caught by another block of code. Printing to users is a last resort; even if you want a user-readable error message, an exception is not the ideal way to deliver one since it is constructed in the guts of a failed task, without the higher level context of what the user was actually trying to do. User error messages ought to be made from information in the exception, combined with higher level knowledge. This is best done in a `catch` block, not a `throw` statement. + + As such, I recommend that you: + + $(LIST + * Don't concatenate error strings at the throw site. Instead, pass the data you would have used to build the string as actual data to the constructor. This lets catchers see the original data without having to try to extract it from a string. For unique data, you will likely need a unique exception type. More on this in the next section. + + * Don't construct error strings in a constructor either, for the same reason. Pass the useful data up the call chain, as exception members, to the maximum extent possible. Exception: if you are passed some data with a temporary lifetime that is important enough to pass up the chain. You may `.idup` or `to!string` to preserve as much data as you can before it is lost, but still store it in a separate member of the Exception subclass object. + + * $(I Do) construct strings out of public members in [getAdditionalPrintableInformation]. When this is called, the user has requested as much relevant information as reasonable in string format. Still, avoid concatenation - it lets you pass as many key/value pairs as you like to the caller. They can concatenate as needed. However, note the words "public members" - everything you do in `getAdditionalPrintableInformation` ought to also be possible for code that caught your exception via your public methods and properties. + ) + + $(H4 Subclasses) + + Any exception with unique data types should be a unique class. Whenever practical, this should be one you write and document at the top-level of a module. But I know we get lazy - me too - and this is why in standard D we'd often fall back to `throw new Exception("some string " ~ some info)`. To help resist these urges, I offer some helper functions to use instead that better achieve the key goal of exceptions - passing structured data up a call chain - while still being convenient to write. + + See: [ArsdException], [Win32Enforce] + ++/ +class ArsdExceptionBase : object.Exception { + /++ + Don't call this except from other exceptions; this is essentially an abstract class. + + Params: + operation = the specific operation that failed, throwing the exception + +/ + package this(string operation, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { + super(operation, file, line, next); + } + + /++ + The toString method will print out several components: + + $(LIST + * The file, line, static message, and object class name from the constructor. You can access these independently with the members `file`, `line`, `msg`, and [printableExceptionName]. + * The generic category codes stored with this exception + * Additional members stored with the exception child classes (e.g. platform error codes, associated function arguments) + * The stack trace associated with the exception. You can access these lines independently with `foreach` over the `info` member. + ) + + This is meant to be read by the developer, not end users. You should wrap your user-relevant tasks in a try/catch block and construct more appropriate error messages from context available there, using the individual properties of the exception to add richness. + +/ + final override void toString(scope void delegate(in char[]) sink) const { + // class name and info from constructor + sink(printableExceptionName); + sink("@"); + sink(file); + sink("("); + char[16] buffer; + sink(intToString(line, buffer[])); + sink("): "); + sink(message); + + getAdditionalPrintableInformation((string name, in char[] value) { + sink("\n"); + sink(name); + sink(": "); + sink(value); + }); + + // full stack trace + sink("\n----------------\n"); + foreach(str; info) { + sink(str); + sink("\n"); + } + } + /// ditto + final override string toString() { + string s; + toString((in char[] chunk) { s ~= chunk; }); + return s; + } + + /++ + Users might like to see additional information with the exception. API consumers should pull this out of properties on your child class, but the parent class might not be able to deal with the arbitrary types at runtime the children can introduce, so bringing them all down to strings simplifies that. + + Overrides should always call `super.getAdditionalPrintableInformation(sink);` before adding additional information by calling the sink with other arguments afterward. + + You should spare no expense in preparing this information - translate error codes, build rich strings, whatever it takes - to make the information here useful to the reader. + +/ + void getAdditionalPrintableInformation(scope void delegate(string name, in char[] value) sink) const { + + } + + /++ + This is the name of the exception class, suitable for printing. This should be static data (e.g. a string literal). Override it in subclasses. + +/ + string printableExceptionName() const { + return typeid(this).name; + } + + /// deliberately hiding `Throwable.msg`. Use [message] and [toString] instead. + @disable final void msg() {} + + override const(char)[] message() const { + return super.msg; + } +} + +/++ + ++/ +class InvalidArgumentsException : ArsdExceptionBase { + static struct InvalidArgument { + string name; + string description; + LimitedVariant givenValue; + } + + InvalidArgument[] invalidArguments; + + this(InvalidArgument[] invalidArguments, string functionName = __PRETTY_FUNCTION__, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { + this.invalidArguments = invalidArguments; + super(functionName, file, line, next); + } + + this(string argumentName, string argumentDescription, LimitedVariant givenArgumentValue = LimitedVariant.init, string functionName = __PRETTY_FUNCTION__, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { + this([ + InvalidArgument(argumentName, argumentDescription, givenArgumentValue) + ], functionName, file, line, next); + } + + override void getAdditionalPrintableInformation(scope void delegate(string name, in char[] value) sink) const { + // FIXME: print the details better + foreach(arg; invalidArguments) + sink("invalidArguments[]", arg.name ~ " " ~ arg.description); + } +} + +/++ + Base class for when you've requested a feature that is not available. It may not be available because it is possible, but not yet implemented, or it might be because it is impossible on your operating system. ++/ +class FeatureUnavailableException : ArsdExceptionBase { + this(string featureName = __PRETTY_FUNCTION__, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { + super(featureName, file, line, next); + } +} + +/++ + This means the feature could be done, but I haven't gotten around to implementing it yet. If you email me, I might be able to add it somewhat quickly and get back to you. ++/ +class NotYetImplementedException : FeatureUnavailableException { + this(string featureName = __PRETTY_FUNCTION__, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { + super(featureName, file, line, next); + } + +} + +/++ + This means the feature is not supported by your current operating system. You might be able to get it in an update, but you might just have to find an alternate way of doing things. ++/ +class NotSupportedException : FeatureUnavailableException { + this(string featureName, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { + super(featureName, file, line, next); + } +} + +/++ + This is a generic exception with attached arguments. It is used when I had to throw something but didn't want to write a new class. + + You can catch an ArsdException to get its passed arguments out. + + You can pass either a base class or a string as `Type`. + + See the examples for how to use it. ++/ +template ArsdException(alias Type, DataTuple...) { + static if(DataTuple.length) + alias Parent = ArsdException!(Type, DataTuple[0 .. $-1]); + else + alias Parent = ArsdExceptionBase; + + class ArsdException : Parent { + DataTuple data; + + this(DataTuple data, string file = __FILE__, size_t line = __LINE__) { + this.data = data; + static if(is(Parent == ArsdExceptionBase)) + super(null, file, line); + else + super(data[0 .. $-1], file, line); + } + + static opCall(R...)(R r, string file = __FILE__, size_t line = __LINE__) { + return new ArsdException!(Type, DataTuple, R)(r, file, line); + } + + override string printableExceptionName() const { + static if(DataTuple.length) + enum str = "ArsdException!(" ~ Type.stringof ~ ", " ~ DataTuple.stringof[1 .. $-1] ~ ")"; + else + enum str = "ArsdException!" ~ Type.stringof; + return str; + } + + override void getAdditionalPrintableInformation(scope void delegate(string name, in char[] value) sink) const { + ArsdExceptionBase.getAdditionalPrintableInformation(sink); + + foreach(idx, datum; data) { + enum int lol = cast(int) idx; + enum key = "[" ~ lol.stringof ~ "] " ~ DataTuple[idx].stringof; + sink(key, toStringInternal(datum)); + } + } + } +} + +/// This example shows how you can throw and catch the ad-hoc exception types. +unittest { + // you can throw and catch by matching the string and argument types + try { + // throw it with parenthesis after the template args (it uses opCall to construct) + throw ArsdException!"Test"(); + // you could also `throw new ArsdException!"test";`, but that gets harder with args + // as we'll see in the following example + assert(0); // remove from docs + } catch(ArsdException!"Test" e) { // catch it without them + // this has no useful information except for the type + // but you can catch it like this and it is still more than generic Exception + } + + // an exception's job is to deliver useful information up the chain + // and you can do that easily by passing arguments: + + try { + throw ArsdException!"Test"(4, "four"); + // you could also `throw new ArsdException!("Test", int, string)(4, "four")` + // but now you start to see how the opCall convenience constructor simplifies things + assert(0); // remove from docs + } catch(ArsdException!("Test", int, string) e) { // catch it and use info by specifying types + assert(e.data[0] == 4); // and extract arguments like this + assert(e.data[1] == "four"); + } + + // a throw site can add additional information without breaking code that catches just some + // generally speaking, each additional argument creates a new subclass on top of the previous args + // so you can cast + + try { + throw ArsdException!"Test"(4, "four", 9); + assert(0); // remove from docs + } catch(ArsdException!("Test", int, string) e) { // this catch still works + assert(e.data[0] == 4); + assert(e.data[1] == "four"); + // but if you were to print it, all the members would be there + // import std.stdio; writeln(e); // would show something like: + /+ + ArsdException!("Test", int, string, int)@file.d(line): + [0] int: 4 + [1] string: four + [2] int: 9 + +/ + // indicating that there's additional information available if you wanted to process it + + // and meanwhile: + ArsdException!("Test", int) e2 = e; // this implicit cast works thanks to the parent-child relationship + ArsdException!"Test" e3 = e; // this works too, the base type/string still matches + + // so catching those types would work too + } +} + +/++ + A tagged union that holds an error code from system apis, meaning one from Windows GetLastError() or C's errno. + + You construct it with `SystemErrorCode(thing)` and the overloaded constructor tags and stores it. ++/ +struct SystemErrorCode { + /// + enum Type { + errno, /// + win32 /// + } + + const Type type; /// + const int code; /// You should technically cast it back to DWORD if it is a win32 code + + /++ + C/unix error are typed as signed ints... + Windows' errors are typed DWORD, aka unsigned... + + so just passing them straight up will pick the right overload here to set the tag. + +/ + this(int errno) { + this.type = Type.errno; + this.code = errno; + } + + /// ditto + this(uint win32) { + this.type = Type.win32; + this.code = win32; + } + + /++ + Returns if the code indicated success. + + Please note that many calls do not actually set a code to success, but rather just don't touch it. Thus this may only be true on `init`. + +/ + bool wasSuccessful() const { + final switch(type) { + case Type.errno: + return this.code == 0; + case Type.win32: + return this.code == 0; + } + } + + /++ + Constructs a string containing both the code and the explanation string. + +/ + string toString() const { + return codeAsString ~ " " ~ errorString; + } + + /++ + The numeric code itself as a string. + + See [errorString] for a text explanation of the code. + +/ + string codeAsString() const { + char[16] buffer; + final switch(type) { + case Type.errno: + return intToString(code, buffer[]).idup; + case Type.win32: + buffer[0 .. 2] = "0x"; + return buffer[0 .. 2 + intToString(code, buffer[2 .. $], IntToStringArgs().withRadix(16).withPadding(8)).length].idup; + } + } + + /++ + A text explanation of the code. See [codeAsString] for a string representation of the numeric representation. + +/ + string errorString() const { + final switch(type) { + case Type.errno: + import core.stdc.string; + auto strptr = strerror(code); + auto orig = strptr; + int len; + while(*strptr++) { + len++; + } + + return orig[0 .. len].idup; + case Type.win32: + version(Windows) { + wchar[256] buffer; + auto size = FormatMessageW( + FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, + null, + code, + MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), + buffer.ptr, + buffer.length, + null + ); + + return makeUtf8StringFromWindowsString(buffer[0 .. size]).stripInternal; + } else { + return null; + } + } + } +} + +/++ + ++/ +struct SavedArgument { + string name; + LimitedVariant value; +} + +/++ + ++/ +class SystemApiException : ArsdExceptionBase { + this(string msg, int originalErrorNo, scope SavedArgument[] args = null, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { + this(msg, SystemErrorCode(originalErrorNo), args, file, line, next); + } + + version(Windows) + this(string msg, DWORD windowsError, scope SavedArgument[] args = null, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { + this(msg, SystemErrorCode(windowsError), args, file, line, next); + } + + this(string msg, SystemErrorCode code, SavedArgument[] args = null, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { + this.errorCode = code; + + // discard stuff that won't fit + if(args.length > this.args.length) + args = args[0 .. this.args.length]; + + this.args[0 .. args.length] = args[]; + + super(msg, file, line, next); + } + + /++ + + +/ + const SystemErrorCode errorCode; + + /++ + + +/ + const SavedArgument[8] args; + + override void getAdditionalPrintableInformation(scope void delegate(string name, in char[] value) sink) const { + super.getAdditionalPrintableInformation(sink); + sink("Error code", errorCode.toString()); + + foreach(arg; args) + if(arg.name !is null) + sink(arg.name, arg.value.toString()); + } + +} + +/++ + The low level use of this would look like `throw new WindowsApiException("MsgWaitForMultipleObjectsEx", GetLastError())` but it is meant to be used from higher level things like [Win32Enforce]. + + History: + Moved from simpledisplay.d to core.d in March 2023 (dub v11.0). ++/ +alias WindowsApiException = SystemApiException; + +/++ + History: + Moved from simpledisplay.d to core.d in March 2023 (dub v11.0). ++/ +alias ErrnoApiException = SystemApiException; + +/++ + Calls the C API function `fn`. If it returns an error value, it throws an [ErrnoApiException] (or subclass) after getting `errno`. ++/ +template ErrnoEnforce(alias fn, alias errorValue = void) { + static if(is(typeof(fn) Return == return)) + static if(is(typeof(fn) Params == __parameters)) { + static if(is(errorValue == void)) { + static if(is(typeof(null) : Return)) + enum errorValueToUse = null; + else static if(is(Return : long)) + enum errorValueToUse = -1; + else + static assert(0, "Please pass the error value"); + } else { + enum errorValueToUse = errorValue; + } + + Return ErrnoEnforce(Params params, ArgSentinel sentinel = ArgSentinel.init, string file = __FILE__, size_t line = __LINE__) { + import core.stdc.errno; + + Return value = fn(params); + + if(value == errorValueToUse) { + SavedArgument[] args; // FIXME + /+ + static foreach(idx; 0 .. Params.length) + args ~= SavedArgument( + __traits(identifier, Params[idx .. idx + 1]), + params[idx] + ); + +/ + throw new ErrnoApiException(__traits(identifier, fn), errno, args, file, line); + } + + return value; + } + } +} + +version(Windows) { + /++ + Calls the Windows API function `fn`. If it returns an error value, it throws a [WindowsApiException] (or subclass) after calling `GetLastError()`. + +/ + template Win32Enforce(alias fn, alias errorValue = void) { + static if(is(typeof(fn) Return == return)) + static if(is(typeof(fn) Params == __parameters)) { + static if(is(errorValue == void)) { + static if(is(Return == BOOL)) + enum errorValueToUse = false; + else static if(is(Return : HANDLE)) + enum errorValueToUse = NULL; + else static if(is(Return == DWORD)) + enum errorValueToUse = cast(DWORD) 0xffffffff; + else + static assert(0, "Please pass the error value"); + } else { + enum errorValueToUse = errorValue; + } + + Return Win32Enforce(Params params, ArgSentinel sentinel = ArgSentinel.init, string file = __FILE__, size_t line = __LINE__) { + Return value = fn(params); + + if(value == errorValueToUse) { + auto error = GetLastError(); + SavedArgument[] args; // FIXME + throw new WindowsApiException(__traits(identifier, fn), error, args, file, line); + } + + return value; + } + } + } + +} + +/+ + =============== + EVENT LOOP CORE + =============== ++/ + +/+ + UI threads + need to get window messages in addition to all the other jobs + I/O Worker threads + need to get commands for read/writes, run them, and send the reply back. not necessary on Windows + if interrupted, check cancel flags. + CPU Worker threads + gets functions, runs them, send reply back. should send a cancel flag to periodically check + Task worker threads + runs fibers and multiplexes them + + + General procedure: + issue the read/write command + if it would block on linux, epoll associate it. otherwise do the callback immediately + + callbacks have default affinity to the current thread, meaning their callbacks always run here + accepts can usually be dispatched to any available thread tho + + // In other words, a single thread can be associated with, at most, one I/O completion port. + + Realistically, IOCP only used if there is no thread affinity. If there is, just do overlapped w/ sleepex. + + + case study: http server + + 1) main thread starts the server. it does an accept loop with no thread affinity. the main thread does NOT check the global queue (the iocp/global epoll) + 2) connections come in and are assigned to first available thread via the iocp/global epoll + 3) these run local event loops until the connection task is finished + + EVENT LOOP TYPES: + 1) main ui thread - MsgWaitForMultipleObjectsEx / epoll on the local ui. it does NOT check the any worker thread thing! + The main ui thread should never terminate until the program is ready to close. + You can have additional ui threads in theory but im not really gonna support that in full; most things will assume there is just the one. simpledisplay's gui thread is the primary if it exists. (and sdpy will prolly continue to be threaded the way it is now) + + The biggest complication is the TerminalDirectToEmulator, where the primary ui thread is NOT the thread that runs `main` + 2) worker thread GetQueuedCompletionStatusEx / epoll on the local thread fd and the global epoll fd + 3) local event loop - check local things only. SleepEx / epoll on local thread fd. This more of a compatibility hack for `waitForCompletion` outside a fiber. + + i'll use: + * QueueUserAPC to send interruptions to a worker thread + * PostQueuedCompletionStatus is to send interruptions to any available thread. + * PostMessage to a window + * ??? to a fiber task + + I also need a way to de-duplicate events in the queue so if you try to push the same thing it won't trigger multiple times.... I might want to keep a duplicate of the thing... really, what I'd do is post the "event wake up" message and keep the queue in my own thing. (WM_PAINT auto-coalesces) + + Destructors need to be able to post messages back to a specific task to queue thread-affinity cleanup. This must be GC safe. + + A task might want to wait on certain events. If the task is a fiber, it yields and gets called upon the event. If the task is a thread, it really has to call the event loop... which can be a loop of loops we want to avoid. `waitForCompletion` is more often gonna be used just to run the loop at top level tho... it might not even check for the global info availability so it'd run the local thing only. + + APCs should not themselves enter an alterable wait cuz it can stack overflow. So generally speaking, they should avoid calling fibers or other event loops. ++/ + +/++ + You can also pass a handle to a specific thread, if you have one. ++/ +enum ThreadToRunIn { + /++ + The callback should be only run by the same thread that set it. + +/ + CurrentThread, + /++ + The UI thread is a special one - it is the supervisor of the workers and the controller of gui and console handles. It is the first thread to call [arsd_core_init] actively running an event loop unless there is a thread that has actively asserted the ui supervisor role. FIXME is this true after i implemen it? + + A ui thread should be always quickly responsive to new events. + + There should only be one main ui thread, in which simpledisplay and minigui can be used. + + Other threads can run like ui threads, but are considered temporary and only concerned with their own needs (it is the default style of loop + for an undeclared thread but will not receive messages from other threads unless there is no other option) + + + Ad-Hoc thread - something running an event loop that isn't another thing + Controller thread - running an explicit event loop instance set as not a task runner or blocking worker + UI thread - simpledisplay's event loop, which it will require remain live for the duration of the program (running two .eventLoops without a parent EventLoop instance will become illegal, throwing at runtime if it happens telling people to change their code + + Windows HANDLES will always be listened on the thread itself that is requesting, UNLESS it is a worker/helper thread, in which case it goes to a coordinator thread. since it prolly can't rely on the parent per se this will have to be one created by arsd core init, UNLESS the parent is inside an explicit EventLoop structure. + + All use the MsgWaitForMultipleObjectsEx pattern + + + +/ + UiThread, + /++ + The callback can be called from any available worker thread. It will be added to a global queue and the first thread to see it will run it. + + These will not run on the UI thread unless there is no other option on the platform (and all platforms this lib supports have other options). + + These are expected to run cooperatively multitasked things; functions that frequently yield as they wait on other tasks. Think a fiber. + + A task runner should be generally responsive to new events. + +/ + AnyAvailableTaskRunnerThread, + /++ + These are expected to run longer blocking, but independent operations. Think an individual function with no context. + + A blocking worker can wait hundreds of milliseconds between checking for new events. + +/ + AnyAvailableBlockingWorkerThread, + /++ + The callback will be duplicated across all threads known to the arsd.core event loop. + + It adds it to an immutable queue that each thread will go through... might just replace with an exit() function. + + + so to cancel all associated tasks for like a web server, it could just have the tasks atomicAdd to a counter and subtract when they are finished. Then you have a single semaphore you signal the number of times you have an active thing and wait for them to acknowledge it. + + threads should report when they start running the loop and they really should report when they terminate but that isn't reliable + + + hmmm what if: all user-created threads (the public api) count as ui threads. only ones created in here are task runners or helpers. ui threads can wait on a global event to exit. + + there's still prolly be one "the" ui thread, which does the handle listening on windows and is the one sdpy wants. + +/ + BroadcastToAllThreads, +} + +/++ + Initializes the arsd core event loop and creates its worker threads. You don't actually have to call this, since the first use of an arsd.core function that requires it will call it implicitly, but calling it yourself gives you a chance to control the configuration more explicitly if you want to. ++/ +void arsd_core_init(int numberOfWorkers = 0) { + +} + +version(Windows) +class WindowsHandleReader_ex { + // Windows handles are always dispatched to the main ui thread, which can then send a command back to a worker thread to run the callback if needed + this(HANDLE handle) {} +} + +version(Posix) +class PosixFdReader_ex { + // posix readers can just register with whatever instance we want to handle the callback +} + +/++ + ++/ +interface ICoreEventLoop { + /++ + Runs the event loop for this thread until the `until` delegate returns `true`. + +/ + final void run(scope bool delegate() until) { + while(!until()) { + runOnce(); + } + } + + /++ + Runs a single iteration of the event loop for this thread. It will return when the first thing happens, but that thing might be totally uninteresting to anyone, or it might trigger significant work you'll wait on. + +/ + void runOnce(); + + // to send messages between threads, i'll queue up a function that just call dispatchMessage. can embed the arg inside the callback helper prolly. + // tho i might prefer to actually do messages w/ run payloads so it is easier to deduplicate i can still dedupe by insepcting the call args so idk + + version(Posix) { + @mustuse + static struct UnregisterToken { + private CoreEventLoopImplementation impl; + private int fd; + private CallbackHelper cb; + + /++ + Unregisters the file descriptor from the event loop and releases the reference to the callback held by the event loop (which will probably free it). + + You must call this when you're done. Normally, this will be right before you close the fd (Which is often after the other side closes it, meaning you got a 0 length read). + +/ + void unregister() { + assert(impl !is null, "Cannot reuse unregister token"); + + version(Arsd_core_epoll) { + impl.unregisterFd(fd); + } else version(Arsd_core_kqueue) { + // intentionally blank - all registrations are one-shot there + // FIXME: actually it might not have gone off yet, in that case we do need to delete the filter + } else static assert(0); + + cb.release(); + this = typeof(this).init; + } + } + + @mustuse + static struct RearmToken { + private bool readable; + private CoreEventLoopImplementation impl; + private int fd; + private CallbackHelper cb; + private uint flags; + + /++ + Calls [UnregisterToken.unregister] + +/ + void unregister() { + assert(impl !is null, "cannot reuse rearm token after unregistering it"); + + version(Arsd_core_epoll) { + impl.unregisterFd(fd); + } else version(Arsd_core_kqueue) { + // intentionally blank - all registrations are one-shot there + // FIXME: actually it might not have gone off yet, in that case we do need to delete the filter + } else static assert(0); + + cb.release(); + this = typeof(this).init; + } + + /++ + Rearms the event so you will get another callback next time it is ready. + +/ + void rearm() { + assert(impl !is null, "cannot reuse rearm token after unregistering it"); + impl.rearmFd(this); + } + } + + UnregisterToken addCallbackOnFdReadable(int fd, CallbackHelper cb); + RearmToken addCallbackOnFdReadableOneShot(int fd, CallbackHelper cb); + RearmToken addCallbackOnFdWritableOneShot(int fd, CallbackHelper cb); + } +} + +/++ + Get the event loop associated with this thread ++/ +ICoreEventLoop getThisThreadEventLoop(EventLoopType type = EventLoopType.AdHoc) { + static ICoreEventLoop loop; + if(loop is null) + loop = new CoreEventLoopImplementation(); + return loop; +} + +/++ + The internal types that will be exposed through other api things. ++/ +package(arsd) enum EventLoopType { + /++ + The event loop is being run temporarily and the thread doesn't promise to keep running it. + +/ + AdHoc, + /++ + The event loop struct has been instantiated at top level. Its destructor will run when the + function exits, which is only at the end of the entire block of work it is responsible for. + + It must be in scope for the whole time the arsd event loop functions are expected to be used + (meaning it should generally be top-level in `main`) + +/ + Explicit, + /++ + A specialization of `Explicit`, so all the same rules apply there, but this is specifically the event loop coming from simpledisplay or minigui. It will run for the duration of the UI's existence. + +/ + Ui, + /++ + A special event loop specifically for threads that listen to the task runner queue and handle I/O events from running tasks. Typically, a task runner runs cooperatively multitasked coroutines (so they prefer not to block the whole thread). + +/ + TaskRunner, + /++ + A special event loop specifically for threads that listen to the helper function request queue. Helper functions are expected to run independently for a somewhat long time (them blocking the thread for some time is normal) and send a reply message back to the requester. + +/ + HelperWorker +} + +/+ + Tasks are given an object to talk to their parent... can be a dialog where it is like + + sendBuffer + waitForWordToProceed + + in a loop + + + Tasks are assigned to a worker thread and may share it with other tasks. ++/ + + +// the GC may not be able to see this! remember, it can be hidden inside kernel buffers +private class CallbackHelper { + import core.memory; + + void call() { + if(callback) + callback(); + } + + void delegate() callback; + void*[3] argsStore; + + void addref() { + atomicOp!"+="(refcount, 1); + } + + void release() { + if(atomicOp!"-="(refcount, 1) <= 0) { + if(flags & 1) + GC.removeRoot(cast(void*) this); + } + } + + private shared(int) refcount; + private uint flags; + + this(void function() callback) { + this( () { callback(); } ); + } + + this(void delegate() callback, bool addRoot = true) { + if(addRoot) { + GC.addRoot(cast(void*) this); + this.flags |= 1; + } + + this.addref(); + this.callback = callback; + } +} + +/++ + This represents a file. Technically, file paths aren't actually strings (for example, on Linux, they need not be valid utf-8, while a D string is supposed to be), even though we almost always use them like that. + + This type is meant to represent a filename / path. I might not keep it around. ++/ +struct FilePath { + string path; + + bool isNull() { + return path is null; + } + + bool opCast(T:bool)() { + return !isNull; + } + + string toString() { + return path; + } + + //alias toString this; +} + +/++ + Represents a generic async, waitable request. ++/ +class AsyncOperationRequest { + /++ + Actually issues the request, starting the operation. + +/ + abstract void start(); + /++ + Cancels the request. This will cause `isComplete` to return true once the cancellation has been processed, but [AsyncOperationResponse.wasSuccessful] will return `false` (unless it completed before the cancellation was processed, in which case it is still allowed to finish successfully). + + After cancelling a request, you should still wait for it to complete to ensure that the task has actually released its resources before doing anything else on it. + + Once a cancellation request has been sent, it cannot be undone. + +/ + abstract void cancel(); + + /++ + Returns `true` if the operation has been completed. It may be completed successfully, cancelled, or have errored out - to check this, call [waitForCompletion] and check the members on the response object. + +/ + abstract bool isComplete(); + /++ + Waits until the request has completed - successfully or otherwise - and returns the response object. It will run an ad-hoc event loop that may call other callbacks while waiting. + + The response object may be embedded in the request object - do not reuse the request until you are finished with the response and do not keep the response around longer than you keep the request. + + + Note to implementers: all subclasses should override this and return their specific response object. You can use the top-level `waitForFirstToCompleteByIndex` function with a single-element static array to help with the implementation. + +/ + abstract AsyncOperationResponse waitForCompletion(); + + /++ + + +/ + // abstract void repeat(); +} + +/++ + ++/ +interface AsyncOperationResponse { + /++ + Returns true if the request completed successfully, finishing what it was supposed to. + + Should be set to `false` if the request was cancelled before completing or encountered an error. + +/ + bool wasSuccessful(); +} + +/++ + It returns the $(I request) so you can identify it more easily. `request.waitForCompletion()` is guaranteed to return the response without any actual wait, since it is already complete when this function returns. + + Please note that "completion" is not necessary successful completion; a request being cancelled or encountering an error also counts as it being completed. + + The `waitForFirstToCompleteByIndex` version instead returns the index of the array entry that completed first. + + It is your responsibility to remove the completed request from the array before calling the function again, since any request already completed will always be immediately returned. + + You might prefer using [asTheyComplete], which will give each request as it completes and loop over until all of them are complete. + + Returns: + `null` or `requests.length` if none completed before returning. ++/ +AsyncOperationRequest waitForFirstToComplete(AsyncOperationRequest[] requests...) { + auto idx = waitForFirstToCompleteByIndex(requests); + if(idx == requests.length) + return null; + return requests[idx]; +} +/// ditto +size_t waitForFirstToCompleteByIndex(AsyncOperationRequest[] requests...) { + size_t helper() { + foreach(idx, request; requests) + if(request.isComplete()) + return idx; + return requests.length; + } + + auto idx = helper(); + // if one is already done, return it + if(idx != requests.length) + return idx; + + // otherwise, run the ad-hoc event loop until one is + // FIXME: what if we are inside a fiber? + auto el = getThisThreadEventLoop(); + el.run(() => (idx = helper()) != requests.length); + + return idx; +} + +/++ + Waits for all the `requests` to complete, giving each one through the range interface as it completes. + + This meant to be used in a foreach loop. + + The `requests` array and its contents must remain valid for the lifetime of the returned range. Its contents may be shuffled as the requests complete (the implementation works through an unstable sort+remove). ++/ +AsTheyCompleteRange asTheyComplete(AsyncOperationRequest[] requests...) { + return AsTheyCompleteRange(requests); +} +/// ditto +struct AsTheyCompleteRange { + AsyncOperationRequest[] requests; + + this(AsyncOperationRequest[] requests) { + this.requests = requests; + + if(requests.length == 0) + return; + + // wait for first one to complete, then move it to the front of the array + moveFirstCompleteToFront(); + } + + private void moveFirstCompleteToFront() { + auto idx = waitForFirstToCompleteByIndex(requests); + + auto tmp = requests[0]; + requests[0] = requests[idx]; + requests[idx] = tmp; + } + + bool empty() { + return requests.length == 0; + } + + void popFront() { + assert(!empty); + /+ + this needs to + 1) remove the front of the array as being already processed (unless it is the initial priming call) + 2) wait for one of them to complete + 3) move the complete one to the front of the array + +/ + + requests[0] = requests[$-1]; + requests = requests[0 .. $-1]; + + if(requests.length) + moveFirstCompleteToFront(); + } + + AsyncOperationRequest front() { + return requests[0]; + } +} + +version(Windows) { + alias NativeFileHandle = HANDLE; /// + alias NativeSocketHandle = SOCKET; /// + alias NativePipeHandle = HANDLE; /// +} else version(Posix) { + alias NativeFileHandle = int; /// + alias NativeSocketHandle = int; /// + alias NativePipeHandle = int; /// +} + +/++ + An `AbstractFile` represents a file handle on the operating system level. You cannot do much with it. ++/ +class AbstractFile { + private { + NativeFileHandle handle; + } + + /++ + +/ + enum OpenMode { + readOnly, /// C's "r", the file is read + writeWithTruncation, /// C's "w", the file is blanked upon opening so it only holds what you write + appendOnly, /// C's "a", writes will always be appended to the file + readAndWrite /// C's "r+", writes will overwrite existing parts of the file based on where you seek (default is at the beginning) + } + + /++ + +/ + enum RequirePreexisting { + no, + yes + } + + /+ + enum SpecialFlags { + randomAccessExpected, /// FILE_FLAG_SEQUENTIAL_SCAN is turned off and posix_fadvise(POSIX_FADV_SEQUENTIAL) + skipCache, /// O_DSYNC, FILE_FLAG_NO_BUFFERING and maybe WRITE_THROUGH. note that metadata still goes through the cache, FlushFileBuffers and fsync can still do those + temporary, /// FILE_ATTRIBUTE_TEMPORARY on Windows, idk how to specify on linux. also FILE_FLAG_DELETE_ON_CLOSE can be combined to make a (almost) all memory file. kinda like a private anonymous mmap i believe. + deleteWhenClosed, /// Windows has a flag for this but idk if it is of any real use + async, /// open it in overlapped mode, all reads and writes must then provide an offset. Only implemented on Windows + } + +/ + + /++ + + +/ + protected this(bool async, FilePath filename, OpenMode mode = OpenMode.readOnly, RequirePreexisting require = RequirePreexisting.no, uint specialFlags = 0) { + version(Windows) { + DWORD access; + DWORD creation; + + final switch(mode) { + case OpenMode.readOnly: + access = GENERIC_READ; + creation = OPEN_EXISTING; + break; + case OpenMode.writeWithTruncation: + access = GENERIC_WRITE; + + final switch(require) { + case RequirePreexisting.no: + creation = CREATE_ALWAYS; + break; + case RequirePreexisting.yes: + creation = TRUNCATE_EXISTING; + break; + } + break; + case OpenMode.appendOnly: + access = FILE_APPEND_DATA; + + final switch(require) { + case RequirePreexisting.no: + creation = CREATE_ALWAYS; + break; + case RequirePreexisting.yes: + creation = OPEN_EXISTING; + break; + } + break; + case OpenMode.readAndWrite: + access = GENERIC_READ | GENERIC_WRITE; + + final switch(require) { + case RequirePreexisting.no: + creation = CREATE_NEW; + break; + case RequirePreexisting.yes: + creation = OPEN_EXISTING; + break; + } + break; + } + + WCharzBuffer wname = WCharzBuffer(filename.path); + + auto handle = CreateFileW( + wname.ptr, + access, + FILE_SHARE_READ, + null, + creation, + FILE_ATTRIBUTE_NORMAL | (async ? FILE_FLAG_OVERLAPPED : 0), + null + ); + + if(handle == INVALID_HANDLE_VALUE) { + // FIXME: throw the filename and other params here too + SavedArgument[3] args; + args[0] = SavedArgument("filename", LimitedVariant(filename.path)); + args[1] = SavedArgument("access", LimitedVariant(access, 2)); + args[2] = SavedArgument("requirePreexisting", LimitedVariant(require == RequirePreexisting.yes)); + throw new WindowsApiException("CreateFileW", GetLastError(), args[]); + } + + this.handle = handle; + } else version(Posix) { + import core.sys.posix.unistd; + import core.sys.posix.fcntl; + + CharzBuffer namez = CharzBuffer(filename.path); + int flags; + + // FIXME does mac not have cloexec for real or is this just a druntime problem????? + version(Arsd_core_has_cloexec) { + flags = O_CLOEXEC; + } else { + scope(success) + setCloExec(this.handle); + } + + if(async) + flags |= O_NONBLOCK; + + final switch(mode) { + case OpenMode.readOnly: + flags |= O_RDONLY; + break; + case OpenMode.writeWithTruncation: + flags |= O_WRONLY | O_TRUNC; + + final switch(require) { + case RequirePreexisting.no: + flags |= O_CREAT; + break; + case RequirePreexisting.yes: + break; + } + break; + case OpenMode.appendOnly: + flags |= O_APPEND; + + final switch(require) { + case RequirePreexisting.no: + flags |= O_CREAT; + break; + case RequirePreexisting.yes: + break; + } + break; + case OpenMode.readAndWrite: + flags |= O_RDWR; + + final switch(require) { + case RequirePreexisting.no: + flags |= O_CREAT; + break; + case RequirePreexisting.yes: + break; + } + break; + } + + auto perms = S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH; + int fd = open(namez.ptr, flags, perms); + if(fd == -1) { + SavedArgument[3] args; + args[0] = SavedArgument("filename", LimitedVariant(filename.path)); + args[1] = SavedArgument("flags", LimitedVariant(flags, 2)); + args[2] = SavedArgument("perms", LimitedVariant(perms, 8)); + throw new ErrnoApiException("open", errno, args[]); + } + + this.handle = fd; + } + } + + /++ + + +/ + private this(NativeFileHandle handleToWrap) { + this.handle = handleToWrap; + } + + // only available on some types of file + long size() { return 0; } + + // note that there is no fsync thing, instead use the special flag. + + /++ + + +/ + void close() { + version(Windows) { + Win32Enforce!CloseHandle(handle); + handle = null; + } else version(Posix) { + import unix = core.sys.posix.unistd; + import core.sys.posix.fcntl; + + ErrnoEnforce!(unix.close)(handle); + handle = -1; + } + } +} + +/++ + ++/ +class File : AbstractFile { + + /++ + Opens a file in synchronous access mode. + + The permission mask is on used on posix systems FIXME: implement it + +/ + this(FilePath filename, OpenMode mode = OpenMode.readOnly, RequirePreexisting require = RequirePreexisting.no, uint specialFlags = 0, uint permMask = 0) { + super(false, filename, mode, require, specialFlags); + } + + /++ + + +/ + ubyte[] read(scope ubyte[] buffer) { + return null; + } + + /++ + + +/ + void write(in void[] buffer) { + } + + enum Seek { + current, + fromBeginning, + fromEnd + } + + // Seeking/telling/sizing is not permitted when appending and some files don't support it + // also not permitted in async mode + void seek(long where, Seek fromWhence) {} + long tell() { return 0; } +} + +/++ + Only one operation can be pending at any time in the current implementation. ++/ +class AsyncFile : AbstractFile { + /++ + Opens a file in asynchronous access mode. + +/ + this(FilePath filename, OpenMode mode = OpenMode.readOnly, RequirePreexisting require = RequirePreexisting.no, uint specialFlags = 0, uint permissionMask = 0) { + // FIXME: implement permissionMask + super(true, filename, mode, require, specialFlags); + } + + package(arsd) this(NativeFileHandle adoptPreSetup) { + super(adoptPreSetup); + } + + /// + AsyncReadRequest read(ubyte[] buffer, long offset = 0) { + return new AsyncReadRequest(this, buffer, offset); + } + + /// + AsyncWriteRequest write(const(void)[] buffer, long offset = 0) { + return new AsyncWriteRequest(this, cast(ubyte[]) buffer, offset); + } + +} + +/++ + Reads or writes a file in one call. It might internally yield, but is generally blocking if it returns values. The callback ones depend on the implementation. + + Tip: prefer the callback ones. If settings where async is possible, it will do async, and if not, it will sync. ++/ +void writeFile(string filename, const(void)[] contents) { + +} + +/// ditto +string readTextFile(string filename, string fileEncoding = null) { + return null; +} + +/// ditto +const(ubyte[]) readBinaryFile(string filename) { + return null; +} + +/+ +private Class recycleObject(Class, Args...)(Class objectToRecycle, Args args) { + if(objectToRecycle is null) + return new Class(args); + // destroy nulls out the vtable which is the first thing in the object + // so if it hasn't already been destroyed, we'll do it here + if((*cast(void**) objectToRecycle) !is null) { + assert(typeid(objectToRecycle) is typeid(Class)); // to make sure we're actually recycling the right kind of object + .destroy(objectToRecycle); + } + + // then go ahead and reinitialize it + ubyte[] rawData = (cast(ubyte*) cast(void*) objectToRecycle)[0 .. __traits(classInstanceSize, Class)]; + rawData[] = (cast(ubyte[]) typeid(Class).initializer)[]; + + objectToRecycle.__ctor(args); + + return objectToRecycle; +} ++/ + +/+ +/++ + Preallocates a class object without initializing it. + + This is suitable *only* for passing to one of the functions in here that takes a preallocated object for recycling. ++/ +Class preallocate(Class)() { + import core.memory; + // FIXME: can i pass NO_SCAN here? + return cast(Class) GC.calloc(__traits(classInstanceSize, Class), 0, typeid(Class)); +} + +OwnedClass!Class preallocateOnStack(Class)() { + +} ++/ + +// thanks for a random person on stack overflow for this function +version(Windows) +BOOL MyCreatePipeEx( + PHANDLE lpReadPipe, + PHANDLE lpWritePipe, + LPSECURITY_ATTRIBUTES lpPipeAttributes, + DWORD nSize, + DWORD dwReadMode, + DWORD dwWriteMode +) +{ + HANDLE ReadPipeHandle, WritePipeHandle; + DWORD dwError; + CHAR[MAX_PATH] PipeNameBuffer; + + if (nSize == 0) { + nSize = 4096; + } + + // FIXME: should be atomic op and gshared + static shared(int) PipeSerialNumber = 0; + + import core.stdc.string; + import core.stdc.stdio; + + sprintf(PipeNameBuffer.ptr, + "\\\\.\\Pipe\\ArsdCoreAnonymousPipe.%08x.%08x".ptr, + GetCurrentProcessId(), + atomicOp!"+="(PipeSerialNumber, 1) + ); + + ReadPipeHandle = CreateNamedPipeA( + PipeNameBuffer.ptr, + 1/*PIPE_ACCESS_INBOUND*/ | dwReadMode, + 0/*PIPE_TYPE_BYTE*/ | 0/*PIPE_WAIT*/, + 1, // Number of pipes + nSize, // Out buffer size + nSize, // In buffer size + 120 * 1000, // Timeout in ms + lpPipeAttributes + ); + + if (! ReadPipeHandle) { + return FALSE; + } + + WritePipeHandle = CreateFileA( + PipeNameBuffer.ptr, + GENERIC_WRITE, + 0, // No sharing + lpPipeAttributes, + OPEN_EXISTING, + FILE_ATTRIBUTE_NORMAL | dwWriteMode, + null // Template file + ); + + if (INVALID_HANDLE_VALUE == WritePipeHandle) { + dwError = GetLastError(); + CloseHandle( ReadPipeHandle ); + SetLastError(dwError); + return FALSE; + } + + *lpReadPipe = ReadPipeHandle; + *lpWritePipe = WritePipeHandle; + return( TRUE ); +} + + + +/+ + + // this is probably useless. + +/++ + Creates a pair of anonymous pipes ready for async operations. + + You can pass some preallocated objects to recycle if you like. ++/ +AsyncAnonymousPipe[2] anonymousPipePair(AsyncAnonymousPipe[2] preallocatedObjects = [null, null], bool inheritable = false) { + version(Posix) { + int[2] fds; + auto ret = pipe(fds); + + if(ret == -1) + throw new SystemApiException("pipe", errno); + + // FIXME: do we want them inheritable? and do we want both sides to be async? + if(!inheritable) { + setCloExec(fds[0]); + setCloExec(fds[1]); + } + // if it is inherited, do we actually want it non-blocking? + makeNonBlocking(fds[0]); + makeNonBlocking(fds[1]); + + return [ + recycleObject(preallocatedObjects[0], fds[0]), + recycleObject(preallocatedObjects[1], fds[1]), + ]; + } else version(Windows) { + HANDLE rp, wp; + // FIXME: do we want them inheritable? and do we want both sides to be async? + if(!MyCreatePipeEx(&rp, &wp, null, 0, FILE_FLAG_OVERLAPPED, FILE_FLAG_OVERLAPPED)) + throw new SystemApiException("MyCreatePipeEx", GetLastError()); + return [ + recycleObject(preallocatedObjects[0], rp), + recycleObject(preallocatedObjects[1], wp), + ]; + } else throw ArsdException!"NotYetImplemented"(); +} + // on posix, just do pipe() w/ non block + // on windows, do an overlapped named pipe server, connect, stop listening, return pair. ++/ + +/+ +class NamedPipe : AsyncFile { + +} ++/ + +/++ + A named pipe ready to accept connections. + + A Windows named pipe is an IPC mechanism usable on local machines or across a Windows network. ++/ +version(Windows) +class NamedPipeServer { + // unix domain socket or windows named pipe + + // Promise!AsyncAnonymousPipe connect; + // Promise!AsyncAnonymousPipe accept; + + // when a new connection arrives, it calls your callback + // can be on a specific thread or on any thread +} + +/++ + Looking these up might be done asynchronously. The objects both represent an async request and its result, which is the actual address the operating system uses. + + When you create an address, it holds a request. You can call `start` and `waitForCompletion` like with other async requests. The request may be run in a helper thread. + + Unlike most the async objects though, its methods will implicitly call `waitForCompletion`. + + Note that The current implementation just blocks. ++/ +class SocketAddress /* : AsyncOperationRequest, AsyncOperationResponse */ { + // maybe accept url? + // unix:///home/me/thing + // ip://0.0.0.0:4555 + // ipv6://[00:00:00:00:00:00] + + // address info + abstract int domain(); + // FIXME: find all cases of this and make sure it is completed first + abstract sockaddr* rawAddr(); + abstract socklen_t rawAddrLength(); + + /+ + // request interface + abstract void start(); + abstract SocketAddress waitForCompletion(); + abstract bool isComplete(); + + // response interface + abstract bool wasSuccessful(); + +/ +} + +/+ +class BluetoothAddress : SocketAddress { + // FIXME it is AF_BLUETOOTH + // see: https://people.csail.mit.edu/albert/bluez-intro/x79.html + // see: https://learn.microsoft.com/en-us/windows/win32/Bluetooth/bluetooth-programming-with-windows-sockets +} ++/ + +version(Posix) // FIXME: find the sockaddr_un definition for Windows too and add it in +final class UnixAddress : SocketAddress { + sockaddr_un address; + + override int domain() { + return AF_UNIX; + } + + override sockaddr* rawAddr() { + return cast(sockaddr*) &address; + } + override socklen_t rawAddrLength() { + return address.sizeof; + } +} + +final class IpAddress : SocketAddress { + sockaddr_in address; + + override int domain() { + return AF_INET; + } + + override sockaddr* rawAddr() { + return cast(sockaddr*) &address; + } + override socklen_t rawAddrLength() { + return address.sizeof; + } +} + +final class Ipv6Address : SocketAddress { + sockaddr_in6 address; + + override int domain() { + return AF_INET6; + } + + override sockaddr* rawAddr() { + return cast(sockaddr*) &address; + } + override socklen_t rawAddrLength() { + return address.sizeof; + } +} + +/++ + For functions that give you an unknown address, you can use this to hold it. ++/ +struct SocketAddressBuffer { + sockaddr address; + socklen_t addrlen; +} + +class AsyncSocket : AsyncFile { + // otherwise: accept, bind, connect, shutdown, close. + + static auto lastError() { + version(Windows) + return WSAGetLastError(); + else + return errno; + } + + static bool wouldHaveBlocked() { + auto error = lastError; + version(Windows) { + return error == WSAEWOULDBLOCK || error == WSAETIMEDOUT; + } else { + return error == EAGAIN || error == EWOULDBLOCK; + } + } + + version(Windows) + enum INVALID = INVALID_SOCKET; + else + enum INVALID = -1; + + // type is mostly SOCK_STREAM or SOCK_DGRAM + /++ + Creates a socket compatible with the given address. It does not actually connect or bind, nor store the address. You will want to pass it again to those functions: + + --- + auto socket = new Socket(address, Socket.Type.Stream); + socket.connect(address).waitForCompletion(); + --- + +/ + this(SocketAddress address, int type, int protocol = 0) { + // need to look up these values for linux + // type |= SOCK_NONBLOCK | SOCK_CLOEXEC; + + handle_ = socket(address.domain(), type, protocol); + if(handle == INVALID) + throw new SystemApiException("socket", lastError()); + + super(cast(NativeFileHandle) handle); // I think that cast is ok on Windows... i think + + version(Posix) { + makeNonBlocking(handle); + setCloExec(handle); + } + + // FIXME: chekc for broadcast + + // FIXME: REUSEADDR ? + + // FIXME: also set NO_DELAY prolly + // int opt = 1; + // setsockopt(handle, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); + } + + /++ + Enabling NODELAY can give latency improvements if you are managing buffers on your end + +/ + void setNoDelay(bool enabled) { + + } + + /++ + + `allowQuickRestart` will set the SO_REUSEADDR on unix and SO_DONTLINGER on Windows, + allowing the application to be quickly restarted despite there still potentially being + pending data in the tcp stack. + + See https://stackoverflow.com/questions/3229860/what-is-the-meaning-of-so-reuseaddr-setsockopt-option-linux for more information. + + If you already set your appropriate socket options or value correctness and reliability of the network stream over restart speed, leave this at the default `false`. + +/ + void bind(SocketAddress address, bool allowQuickRestart = false) { + if(allowQuickRestart) { + // FIXME + } + + auto ret = .bind(handle, address.rawAddr, address.rawAddrLength); + if(ret == -1) + throw new SystemApiException("bind", lastError); + } + + /++ + You must call [bind] before this. + + The backlog should be set to a value where your application can reliably catch up on the backlog in a reasonable amount of time under average load. It is meant to smooth over short duration bursts and making it too big will leave clients hanging - which might cause them to try to reconnect, thinking things got lost in transit, adding to your impossible backlog. + + I personally tend to set this to be two per worker thread unless I have actual real world measurements saying to do something else. It is a bit arbitrary and not based on legitimate reasoning, it just seems to work for me (perhaps just because it has never really been put to the test). + +/ + void listen(int backlog) { + auto ret = .listen(handle, backlog); + if(ret == -1) + throw new SystemApiException("listen", lastError); + } + + /++ + +/ + void shutdown(int how) { + auto ret = .shutdown(handle, how); + if(ret == -1) + throw new SystemApiException("shutdown", lastError); + } + + /++ + +/ + override void close() { + version(Windows) + closesocket(handle); + else + .close(handle); + handle_ = -1; + } + + /++ + You can also construct your own request externally to control the memory more. + +/ + AsyncConnectRequest connect(SocketAddress address) { + return new AsyncConnectRequest(this, address); + } + + /++ + You can also construct your own request externally to control the memory more. + +/ + AsyncAcceptRequest accept() { + return new AsyncAcceptRequest(this); + } + + // note that send is just sendto w/ a null address + // and receive is just receivefrom w/ a null address + /++ + You can also construct your own request externally to control the memory more. + +/ + AsyncSendRequest send(const(ubyte)[] buffer, int flags = 0) { + return new AsyncSendRequest(this, buffer, null, flags); + } + + /++ + You can also construct your own request externally to control the memory more. + +/ + AsyncReceiveRequest receive(ubyte[] buffer, int flags = 0) { + return new AsyncReceiveRequest(this, buffer, null, flags); + } + + /++ + You can also construct your own request externally to control the memory more. + +/ + AsyncSendRequest sendTo(const(ubyte)[] buffer, SocketAddress address, int flags = 0) { + return new AsyncSendRequest(this, buffer, address, flags); + } + /++ + You can also construct your own request externally to control the memory more. + +/ + AsyncReceiveRequest receiveFrom(ubyte[] buffer, SocketAddressBuffer* address, int flags = 0) { + return new AsyncReceiveRequest(this, buffer, address, flags); + } + + /++ + +/ + SocketAddress localAddress() { + return null; // FIXME + } + /++ + +/ + SocketAddress peerAddress() { + return null; // FIXME + } + + // for unix sockets on unix only: send/receive fd, get peer creds + + /++ + + +/ + final NativeSocketHandle handle() { + return handle_; + } + + private NativeSocketHandle handle_; +} + +/++ ++/ +class AsyncConnectRequest : AsyncOperationRequest { + this(AsyncSocket socket, SocketAddress address) { + + } + + override void start() {} + override void cancel() {} + override bool isComplete() { return true; } + override AsyncConnectResponse waitForCompletion() { assert(0); } +} +/++ ++/ +class AsyncConnectResponse : AsyncOperationResponse { + const SystemErrorCode errorCode; + + this(SystemErrorCode errorCode) { + this.errorCode = errorCode; + } + + override bool wasSuccessful() { + return errorCode.wasSuccessful; + } + +} + +/++ ++/ +class AsyncAcceptRequest : AsyncOperationRequest { + this(AsyncSocket socket) { + + } + + override void start() {} + override void cancel() {} + override bool isComplete() { return true; } + override AsyncConnectResponse waitForCompletion() { assert(0); } +} +/++ ++/ +class AsyncAcceptResponse : AsyncOperationResponse { + AsyncSocket newSocket; + const SystemErrorCode errorCode; + + this(AsyncSocket newSocket, SystemErrorCode errorCode) { + this.newSocket = newSocket; + this.errorCode = errorCode; + } + + override bool wasSuccessful() { + return errorCode.wasSuccessful; + } +} + +/++ ++/ +class AsyncReceiveRequest : AsyncOperationRequest { + struct LowLevelOperation { + AsyncSocket file; + ubyte[] buffer; + int flags; + SocketAddressBuffer* address; + + this(typeof(this.tupleof) args) { + this.tupleof = args; + } + + version(Windows) { + auto opCall(OVERLAPPED* overlapped, LPOVERLAPPED_COMPLETION_ROUTINE ocr) { + WSABUF buf; + buf.len = cast(int) buffer.length; + buf.buf = cast(typeof(buf.buf)) buffer.ptr; + + uint flags = this.flags; + + if(address is null) + return WSARecv(file.handle, &buf, 1, null, &flags, overlapped, ocr); + else { + return WSARecvFrom(file.handle, &buf, 1, null, &flags, &(address.address), &(address.addrlen), overlapped, ocr); + } + } + } else { + auto opCall() { + if(address is null) + return core.sys.posix.sys.socket.recv(file.handle, buffer.ptr, buffer.length, flags); + else + return core.sys.posix.sys.socket.recvfrom(file.handle, buffer.ptr, buffer.length, flags, &(address.address), &(address.addrlen)); + } + } + + string errorString() { + return "Receive"; + } + } + mixin OverlappedIoRequest!(AsyncReceiveResponse, LowLevelOperation); + + this(AsyncSocket socket, ubyte[] buffer, SocketAddressBuffer* address, int flags) { + llo = LowLevelOperation(socket, buffer, flags, address); + this.response = typeof(this.response).defaultConstructed; + } + +} +/++ ++/ +class AsyncReceiveResponse : AsyncOperationResponse { + const ubyte[] bufferWritten; + const SystemErrorCode errorCode; + + this(SystemErrorCode errorCode, const(ubyte)[] bufferWritten) { + this.errorCode = errorCode; + this.bufferWritten = bufferWritten; + } + + override bool wasSuccessful() { + return errorCode.wasSuccessful; + } +} + +/++ ++/ +class AsyncSendRequest : AsyncOperationRequest { + struct LowLevelOperation { + AsyncSocket file; + const(ubyte)[] buffer; + int flags; + SocketAddress address; + + this(typeof(this.tupleof) args) { + this.tupleof = args; + } + + version(Windows) { + auto opCall(OVERLAPPED* overlapped, LPOVERLAPPED_COMPLETION_ROUTINE ocr) { + WSABUF buf; + buf.len = cast(int) buffer.length; + buf.buf = cast(typeof(buf.buf)) buffer.ptr; + + if(address is null) + return WSASend(file.handle, &buf, 1, null, flags, overlapped, ocr); + else { + return WSASendTo(file.handle, &buf, 1, null, flags, address.rawAddr, address.rawAddrLength, overlapped, ocr); + } + } + } else { + auto opCall() { + if(address is null) + return core.sys.posix.sys.socket.send(file.handle, buffer.ptr, buffer.length, flags); + else + return core.sys.posix.sys.socket.sendto(file.handle, buffer.ptr, buffer.length, flags, address.rawAddr, address.rawAddrLength); + } + } + + string errorString() { + return "Send"; + } + } + mixin OverlappedIoRequest!(AsyncSendResponse, LowLevelOperation); + + this(AsyncSocket socket, const(ubyte)[] buffer, SocketAddress address, int flags) { + llo = LowLevelOperation(socket, buffer, flags, address); + this.response = typeof(this.response).defaultConstructed; + } +} + +/++ ++/ +class AsyncSendResponse : AsyncOperationResponse { + const ubyte[] bufferWritten; + const SystemErrorCode errorCode; + + this(SystemErrorCode errorCode, const(ubyte)[] bufferWritten) { + this.errorCode = errorCode; + this.bufferWritten = bufferWritten; + } + + override bool wasSuccessful() { + return errorCode.wasSuccessful; + } + +} + +/++ + A socket bound and ready to accept connections. + + Depending on the specified address, it can be tcp, tcpv6, or unix domain. ++/ +class StreamServer { + this(SocketAddress listenTo) { + + } + // when a new connection arrives, it calls your callback + // can be on a specific thread or on any thread +} + +/++ + A socket bound and ready to use receiveFrom + + Depending on the address, it can be udp or unix domain. ++/ +class DatagramListener { + // whenever a udp message arrives, it calls your callback + // can be on a specific thread or on any thread + + // UDP is realistically just an async read on the bound socket + // just it can get the "from" data out and might need the "more in packet" flag +} + +/++ + Just in case I decide to change the implementation some day. ++/ +alias AsyncAnonymousPipe = AsyncFile; + + +// AsyncAnonymousPipe connectNamedPipe(AsyncAnonymousPipe preallocated, string name) + +// unix fifos are considered just non-seekable files and have no special support in the lib; open them as a regular file w/ the async flag. + +// DIRECTORY LISTINGS + // not async, so if you want that, do it in a helper thread + // just a convenient function to have (tho phobos has a decent one too, importing it expensive af) + +/++ + Note that the order of items called for your delegate is undefined; if you want it sorted, you'll have to collect and sort yourself. But it *might* be sorted by the OS (on Windows, it almost always is), so consider that when choosing a sorting algorithm. + + History: + previously in minigui as a private function. Moved to arsd.core on April 3, 2023 ++/ +GetFilesResult getFiles(string directory, scope void delegate(string name, bool isDirectory) dg) { + // FIXME: my buffers here aren't great lol + + SavedArgument[1] argsForException() { + return [ + SavedArgument("directory", LimitedVariant(directory)), + ]; + } + + version(Windows) { + WIN32_FIND_DATA data; + // FIXME: if directory ends with / or \\ ? + WCharzBuffer search = WCharzBuffer(directory ~ "/*"); + auto handle = FindFirstFileW(search.ptr, &data); + scope(exit) if(handle !is INVALID_HANDLE_VALUE) FindClose(handle); + if(handle is INVALID_HANDLE_VALUE) { + if(GetLastError() == ERROR_FILE_NOT_FOUND) + return GetFilesResult.fileNotFound; + throw new WindowsApiException("FindFirstFileW", GetLastError(), argsForException()[]); + } + + try_more: + + string name = makeUtf8StringFromWindowsString(data.cFileName[0 .. findIndexOfZero(data.cFileName[])]); + + dg(name, (data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) ? true : false); + + auto ret = FindNextFileW(handle, &data); + if(ret == 0) { + if(GetLastError() == ERROR_NO_MORE_FILES) + return GetFilesResult.success; + throw new WindowsApiException("FindNextFileW", GetLastError(), argsForException()[]); + } + + goto try_more; + + } else version(Posix) { + import core.sys.posix.dirent; + import core.stdc.errno; + auto dir = opendir((directory ~ "\0").ptr); + scope(exit) + if(dir) closedir(dir); + if(dir is null) + throw new ErrnoApiException("opendir", errno, argsForException()); + + auto dirent = readdir(dir); + if(dirent is null) + return GetFilesResult.fileNotFound; + + try_more: + + string name = dirent.d_name[0 .. findIndexOfZero(dirent.d_name[])].idup; + + dg(name, dirent.d_type == DT_DIR); + + dirent = readdir(dir); + if(dirent is null) + return GetFilesResult.success; + + goto try_more; + } else static assert(0); +} + +/// ditto +enum GetFilesResult { + success, + fileNotFound +} + +/++ + This is currently a simplified glob where only the * wildcard in the first or last position gets special treatment or a single * in the middle. + + More things may be added later to be more like what Phobos supports. ++/ +bool matchesFilePattern(scope const(char)[] name, scope const(char)[] pattern) { + if(pattern.length == 0) + return false; + if(pattern == "*") + return true; + if(pattern.length > 2 && pattern[0] == '*' && pattern[$-1] == '*') { + // if the rest of pattern appears in name, it is good + return name.indexOf(pattern[1 .. $-1]) != -1; + } else if(pattern[0] == '*') { + // if the rest of pattern is at end of name, it is good + return name.endsWith(pattern[1 .. $]); + } else if(pattern[$-1] == '*') { + // if the rest of pattern is at start of name, it is good + return name.startsWith(pattern[0 .. $-1]); + } else if(pattern.length >= 3) { + auto idx = pattern.indexOf("*"); + if(idx != -1) { + auto lhs = pattern[0 .. idx]; + auto rhs = pattern[idx + 1 .. $]; + if(name.length >= lhs.length + rhs.length) { + return name.startsWith(lhs) && name.endsWith(rhs); + } else { + return false; + } + } + } + + return name == pattern; +} + +unittest { + assert("test.html".matchesFilePattern("*")); + assert("test.html".matchesFilePattern("*.html")); + assert("test.html".matchesFilePattern("*.*")); + assert("test.html".matchesFilePattern("test.*")); + assert(!"test.html".matchesFilePattern("pest.*")); + assert(!"test.html".matchesFilePattern("*.dhtml")); + + assert("test.html".matchesFilePattern("t*.html")); + assert(!"test.html".matchesFilePattern("e*.html")); +} + +package(arsd) int indexOf(scope const(char)[] haystack, scope const(char)[] needle) { + if(haystack.length < needle.length) + return -1; + if(haystack == needle) + return 0; + foreach(i; 0 .. haystack.length - needle.length + 1) + if(haystack[i .. i + needle.length] == needle) + return cast(int) i; + return -1; +} + +unittest { + assert("foo".indexOf("f") == 0); + assert("foo".indexOf("o") == 1); + assert("foo".indexOf("foo") == 0); + assert("foo".indexOf("oo") == 1); + assert("foo".indexOf("fo") == 0); + assert("foo".indexOf("boo") == -1); + assert("foo".indexOf("food") == -1); +} + +package(arsd) bool endsWith(scope const(char)[] haystack, scope const(char)[] needle) { + if(needle.length > haystack.length) + return false; + return haystack[$ - needle.length .. $] == needle; +} + +unittest { + assert("foo".endsWith("o")); + assert("foo".endsWith("oo")); + assert("foo".endsWith("foo")); + assert(!"foo".endsWith("food")); + assert(!"foo".endsWith("d")); +} + +package(arsd) bool startsWith(scope const(char)[] haystack, scope const(char)[] needle) { + if(needle.length > haystack.length) + return false; + return haystack[0 .. needle.length] == needle; +} + +unittest { + assert("foo".startsWith("f")); + assert("foo".startsWith("fo")); + assert("foo".startsWith("foo")); + assert(!"foo".startsWith("food")); + assert(!"foo".startsWith("d")); +} + + +// FILE/DIR WATCHES + // linux does it by name, windows and bsd do it by handle/descriptor + // dispatches change event to either your thread or maybe the any task` queue. + +/++ + ++/ +class DirectoryWatcher { + private { + version(Arsd_core_windows) { + OVERLAPPED overlapped; + HANDLE hDirectory; + ubyte[] buffer; + + extern(Windows) + static void overlappedCompletionRoutine(DWORD dwErrorCode, DWORD dwNumberOfBytesTransferred, LPOVERLAPPED lpOverlapped) { + typeof(this) rr = cast(typeof(this)) (cast(void*) lpOverlapped - typeof(this).overlapped.offsetof); + + // dwErrorCode + auto response = rr.buffer[0 .. dwNumberOfBytesTransferred]; + + while(response.length) { + auto fni = cast(FILE_NOTIFY_INFORMATION*) response.ptr; + auto filename = fni.FileName[0 .. fni.FileNameLength]; + + if(fni.NextEntryOffset) + response = response[fni.NextEntryOffset .. $]; + else + response = response[$..$]; + + // FIXME: I think I need to pin every overlapped op while it is pending + // and unpin it when it is returned. GC.addRoot... but i don't wanna do that + // every op so i guess i should do a refcount scheme similar to the other callback helper. + + rr.changeHandler( + FilePath(makeUtf8StringFromWindowsString(filename)), // FIXME: this is a relative path + ChangeOperation.unknown // FIXME this is fni.Action + ); + } + + rr.requestRead(); + } + + void requestRead() { + DWORD ignored; + if(!ReadDirectoryChangesW( + hDirectory, + buffer.ptr, + cast(int) buffer.length, + recursive, + FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_CREATION | FILE_NOTIFY_CHANGE_FILE_NAME, + &ignored, + &overlapped, + &overlappedCompletionRoutine + )) { + auto error = GetLastError(); + /+ + if(error == ERROR_IO_PENDING) { + // not expected here, the docs say it returns true when queued + } + +/ + + throw new SystemApiException("ReadDirectoryChangesW", error); + } + } + } else version(Arsd_core_epoll) { + static int inotifyfd = -1; // this is TLS since it is associated with the thread's event loop + static ICoreEventLoop.UnregisterToken inotifyToken; + static CallbackHelper inotifycb; + static DirectoryWatcher[int] watchMappings; + + static ~this() { + if(inotifyfd != -1) { + close(inotifyfd); + inotifyfd = -1; + } + } + + import core.sys.linux.sys.inotify; + + int watchId = -1; + + static void inotifyReady() { + // read from it + ubyte[256 /* NAME_MAX + 1 */ + inotify_event.sizeof] sbuffer; + + auto ret = read(inotifyfd, sbuffer.ptr, sbuffer.length); + if(ret == -1) { + auto errno = errno; + if(errno == EAGAIN || errno == EWOULDBLOCK) + return; + throw new SystemApiException("read inotify", errno); + } else if(ret == 0) { + assert(0, "I don't think this is ever supposed to happen"); + } + + auto buffer = sbuffer[0 .. ret]; + + while(buffer.length > 0) { + inotify_event* event = cast(inotify_event*) buffer.ptr; + buffer = buffer[inotify_event.sizeof .. $]; + char[] filename = cast(char[]) buffer[0 .. event.len]; + buffer = buffer[event.len .. $]; + + // note that filename is padded with zeroes, so it is actually a stringz + + if(auto obj = event.wd in watchMappings) { + (*obj).changeHandler( + FilePath(stringz(filename.ptr).borrow.idup), // FIXME: this is a relative path + ChangeOperation.unknown // FIXME + ); + } else { + // it has probably already been removed + } + } + } + } else version(Arsd_core_kqueue) { + int fd; + CallbackHelper cb; + } + + FilePath path; + string globPattern; + bool recursive; + void delegate(FilePath filename, ChangeOperation op) changeHandler; + } + + enum ChangeOperation { + unknown, + deleted, // NOTE_DELETE, IN_DELETE, FILE_NOTIFY_CHANGE_FILE_NAME + written, // NOTE_WRITE / NOTE_EXTEND / NOTE_TRUNCATE, IN_MODIFY, FILE_NOTIFY_CHANGE_LAST_WRITE / FILE_NOTIFY_CHANGE_SIZE + renamed, // NOTE_RENAME, the moved from/to in linux, FILE_NOTIFY_CHANGE_FILE_NAME + metadataChanged // NOTE_ATTRIB, IN_ATTRIB, FILE_NOTIFY_CHANGE_ATTRIBUTES + + // there is a NOTE_OPEN on freebsd 13, and the access change on Windows. and an open thing on linux. so maybe i can do note open/note_read too. + } + + /+ + Windows and Linux work best when you watch directories. The operating system tells you the name of files as they change. + + BSD doesn't support this. You can only get names and reports when a file is modified by watching specific files. AS such, when you watch a directory on those systems, your delegate will be called with a null path. Cross-platform applications should check for this and not assume the name is always usable. + + inotify is kinda clearly the best of the bunch, with Windows in second place, and kqueue dead last. + + + If path to watch is a directory, it signals when a file inside the directory (only one layer deep) is created or modified. This is the most efficient on Windows and Linux. + + If a path is a file, it only signals when that specific file is written. This is most efficient on BSD. + + + The delegate is called when something happens. Note that the path modified may not be accurate on all systems when you are watching a directory. + +/ + + /++ + Watches a directory and its contents. If the `globPattern` is `null`, it will not attempt to add child items but also will not filter it, meaning you will be left with platform-specific behavior. + + On Windows, the globPattern is just used to filter events. + + On Linux, the `recursive` flag, if set, will cause it to add additional OS-level watches for each subdirectory. + + On BSD, anything other than a null pattern will cause a directory scan to add files to the watch list. + + For best results, use the most limited thing you need, as watches can get quite involved on the bsd systems. + + Newly added files and subdirectories may not be automatically added in all cases, meaning if it is added and then subsequently modified, you might miss a notification. + + If the event queue is too busy, the OS may skip a notification. + + You should always offer some way for the user to force a refresh and not rely on notifications being present; they are a convenience when they work, not an always reliable method. + +/ + this(FilePath directoryToWatch, string globPattern, bool recursive, void delegate(FilePath pathModified, ChangeOperation op) dg) { + this.path = directoryToWatch; + this.globPattern = globPattern; + this.recursive = recursive; + this.changeHandler = dg; + + version(Arsd_core_windows) { + WCharzBuffer wname = directoryToWatch.path; + buffer = new ubyte[](1024); + hDirectory = CreateFileW( + wname.ptr, + GENERIC_READ, + FILE_SHARE_READ, + null, + OPEN_EXISTING, + FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED | FILE_FLAG_BACKUP_SEMANTICS, + null + ); + if(hDirectory == INVALID_HANDLE_VALUE) + throw new SystemApiException("CreateFileW", GetLastError()); + + requestRead(); + } else version(Arsd_core_epoll) { + auto el = getThisThreadEventLoop(); + + // no need for sync because it is thread-local + if(inotifyfd == -1) { + inotifyfd = inotify_init1(IN_NONBLOCK | IN_CLOEXEC); + if(inotifyfd == -1) + throw new SystemApiException("inotify_init1", errno); + + inotifycb = new CallbackHelper(&inotifyReady); + inotifyToken = el.addCallbackOnFdReadable(inotifyfd, inotifycb); + } + + uint event_mask = IN_CREATE | IN_MODIFY | IN_DELETE; // FIXME + CharzBuffer dtw = directoryToWatch.path; + auto watchId = inotify_add_watch(inotifyfd, dtw.ptr, event_mask); + if(watchId < -1) + throw new SystemApiException("inotify_add_watch", errno, [SavedArgument("path", LimitedVariant(directoryToWatch.path))]); + + watchMappings[watchId] = this; + + // FIXME: recursive needs to add child things individually + + } else version(Arsd_core_kqueue) { + auto el = cast(CoreEventLoopImplementation) getThisThreadEventLoop(); + + // FIXME: need to scan for globPattern + // when a new file is added, i'll have to diff my list to detect it and open it too + // and recursive might need to scan down too. + + kevent_t ev; + + import core.sys.posix.fcntl; + CharzBuffer buffer = CharzBuffer(directoryToWatch.path); + fd = ErrnoEnforce!open(buffer.ptr, O_RDONLY); + setCloExec(fd); + + cb = new CallbackHelper(&triggered); + + EV_SET(&ev, fd, EVFILT_VNODE, EV_ADD | EV_ENABLE | EV_CLEAR, NOTE_WRITE, 0, cast(void*) cb); + ErrnoEnforce!kevent(el.kqueuefd, &ev, 1, null, 0, null); + } else assert(0, "Not yet implemented for this platform"); + } + + private void triggered() { + writeln("triggered"); + } + + void dispose() { + version(Arsd_core_windows) { + CloseHandle(hDirectory); + } else version(Arsd_core_epoll) { + watchMappings.remove(watchId); // I could also do this on the IN_IGNORE notification but idk + inotify_rm_watch(inotifyfd, watchId); + } else version(Arsd_core_kqueue) { + ErrnoEnforce!close(fd); + fd = -1; + } + } +} + +version(none) +void main() { + + // auto file = new AsyncFile(FilePath("test.txt"), AsyncFile.OpenMode.writeWithTruncation, AsyncFile.RequirePreexisting.yes); + + /+ + getFiles("c:/windows\\", (string filename, bool isDirectory) { + writeln(filename, " ", isDirectory ? "[dir]": "[file]"); + }); + +/ + + auto w = new DirectoryWatcher(FilePath("."), "*", false, (path, op) { + writeln(path.path); + }); + getThisThreadEventLoop().run(() => false); +} + +/++ + This starts up a local pipe. If it is already claimed, it just communicates with the existing one through the interface. ++/ +class SingleInstanceApplication { + // FIXME +} + +version(none) +void main() { + + auto file = new AsyncFile(FilePath("test.txt"), AsyncFile.OpenMode.writeWithTruncation, AsyncFile.RequirePreexisting.yes); + + auto buffer = cast(ubyte[]) "hello"; + auto wr = new AsyncWriteRequest(file, buffer, 0); + wr.start(); + + wr.waitForCompletion(); + + file.close(); +} + +/++ + Implementation details of some requests. You shouldn't need to know any of this, the interface is all public. ++/ +mixin template OverlappedIoRequest(Response, LowLevelOperation) { + private { + LowLevelOperation llo; + + OwnedClass!Response response; + + version(Windows) { + OVERLAPPED overlapped; + + extern(Windows) + static void overlappedCompletionRoutine(DWORD dwErrorCode, DWORD dwNumberOfBytesTransferred, LPOVERLAPPED lpOverlapped) { + typeof(this) rr = cast(typeof(this)) (cast(void*) lpOverlapped - typeof(this).overlapped.offsetof); + + rr.response = typeof(rr.response)(SystemErrorCode(dwErrorCode), rr.llo.buffer[0 .. dwNumberOfBytesTransferred]); + rr.state_ = State.complete; + + // FIXME: on complete? + + // this will queue our CallbackHelper and that should be run at the end of the event loop after it is woken up by the APC run + } + } + + version(Posix) { + ICoreEventLoop.RearmToken eventRegistration; + CallbackHelper cb; + + final CallbackHelper getCb() { + if(cb is null) + cb = new CallbackHelper(&cbImpl); + return cb; + } + + final void cbImpl() { + // it is ready to complete, time to do it + auto ret = llo(); + markCompleted(ret, errno); + } + + void markCompleted(long ret, int errno) { + // maybe i should queue an apc to actually do it, to ensure the event loop has cycled... FIXME + if(ret == -1) + response = typeof(response)(SystemErrorCode(errno), null); + else + response = typeof(response)(SystemErrorCode(0), llo.buffer[0 .. cast(size_t) ret]); + state_ = State.complete; + } + } + } + + enum State { + unused, + started, + inProgress, + complete + } + private State state_; + + override void start() { + assert(state_ == State.unused); + + state_ = State.started; + + version(Windows) { + if(llo(&overlapped, &overlappedCompletionRoutine)) { + // all good, though GetLastError() might have some informative info + } else { + // operation failed, the operation is always ReadFileEx or WriteFileEx so it won't give the io pending thing here + // should i issue error async? idk + state_ = State.complete; + throw new SystemApiException(llo.errorString(), GetLastError()); + } + + // ReadFileEx always queues, even if it completed synchronously. I *could* check the get overlapped result and sleepex here but i'm prolly better off just letting the event loop do its thing anyway. + } else version(Posix) { + + // first try to just do it + auto ret = llo(); + + auto errno = errno; + if(ret == -1 && (errno == EAGAIN || errno == EWOULDBLOCK)) { // unable to complete right now, register and try when it is ready + eventRegistration = getThisThreadEventLoop().addCallbackOnFdReadableOneShot(this.llo.file.handle, this.getCb); + } else { + // i could set errors sync or async and since it couldn't even start, i think a sync exception is the right way + if(ret == -1) + throw new SystemApiException(llo.errorString(), errno); + markCompleted(ret, errno); // it completed synchronously (if it is an error nor not is handled by the completion handler) + } + } + } + + + override void cancel() { + if(state_ == State.complete) + return; // it has already finished, just leave it alone, no point discarding what is already done + version(Windows) { + if(state_ != State.unused) + Win32Enforce!CancelIoEx(llo.file.AbstractFile.handle, &overlapped); + // Windows will notify us when the cancellation is complete, so we need to wait for that before updating the state + } else version(Posix) { + if(state_ != State.unused) + eventRegistration.unregister(); + markCompleted(-1, ECANCELED); + } + } + + override bool isComplete() { + // just always let the event loop do it instead + return state_ == State.complete; + + /+ + version(Windows) { + return HasOverlappedIoCompleted(&overlapped); + } else version(Posix) { + return state_ == State.complete; + + } + +/ + } + + override Response waitForCompletion() { + if(state_ == State.unused) + start(); + + // FIXME: if we are inside a fiber, we can set a oncomplete callback and then yield instead... + if(state_ != State.complete) + getThisThreadEventLoop().run(&isComplete); + + /+ + version(Windows) { + SleepEx(INFINITE, true); + + //DWORD numberTransferred; + //Win32Enforce!GetOverlappedResult(file.handle, &overlapped, &numberTransferred, true); + } else version(Posix) { + getThisThreadEventLoop().run(&isComplete); + } + +/ + + return response; + } +} + +/++ + You can write to a file asynchronously by creating one of these. ++/ +final class AsyncWriteRequest : AsyncOperationRequest { + struct LowLevelOperation { + AsyncFile file; + ubyte[] buffer; + long offset; + + this(typeof(this.tupleof) args) { + this.tupleof = args; + } + + version(Windows) { + auto opCall(OVERLAPPED* overlapped, LPOVERLAPPED_COMPLETION_ROUTINE ocr) { + overlapped.Offset = (cast(ulong) offset) & 0xffff_ffff; + overlapped.OffsetHigh = ((cast(ulong) offset) >> 32) & 0xffff_ffff; + return WriteFileEx(file.handle, buffer.ptr, cast(int) buffer.length, overlapped, ocr); + } + } else { + auto opCall() { + return core.sys.posix.unistd.write(file.handle, buffer.ptr, buffer.length); + } + } + + string errorString() { + return "Write"; + } + } + mixin OverlappedIoRequest!(AsyncWriteResponse, LowLevelOperation); + + this(AsyncFile file, ubyte[] buffer, long offset) { + this.llo = LowLevelOperation(file, buffer, offset); + response = typeof(response).defaultConstructed; + } +} + +/++ + ++/ +class AsyncWriteResponse : AsyncOperationResponse { + const ubyte[] bufferWritten; + const SystemErrorCode errorCode; + + this(SystemErrorCode errorCode, const(ubyte)[] bufferWritten) { + this.errorCode = errorCode; + this.bufferWritten = bufferWritten; + } + + override bool wasSuccessful() { + return errorCode.wasSuccessful; + } +} + +/++ + ++/ +final class AsyncReadRequest : AsyncOperationRequest { + struct LowLevelOperation { + AsyncFile file; + ubyte[] buffer; + long offset; + + this(typeof(this.tupleof) args) { + this.tupleof = args; + } + + version(Windows) { + auto opCall(OVERLAPPED* overlapped, LPOVERLAPPED_COMPLETION_ROUTINE ocr) { + overlapped.Offset = (cast(ulong) offset) & 0xffff_ffff; + overlapped.OffsetHigh = ((cast(ulong) offset) >> 32) & 0xffff_ffff; + return ReadFileEx(file.handle, buffer.ptr, cast(int) buffer.length, overlapped, ocr); + } + } else { + auto opCall() { + return core.sys.posix.unistd.read(file.handle, buffer.ptr, buffer.length); + } + } + + string errorString() { + return "Read"; + } + } + mixin OverlappedIoRequest!(AsyncReadResponse, LowLevelOperation); + + /++ + The file must have the overlapped flag enabled on Windows and the nonblock flag set on Posix. + + The buffer MUST NOT be touched by you - not used by another request, modified, read, or freed, including letting a static array going out of scope - until this request's `isComplete` returns `true`. + + The offset is where to start reading a disk file. For all other types of files, pass 0. + +/ + this(AsyncFile file, ubyte[] buffer, long offset) { + this.llo = LowLevelOperation(file, buffer, offset); + response = typeof(response).defaultConstructed; + } + + /++ + + +/ + // abstract void repeat(); +} + +/++ + ++/ +class AsyncReadResponse : AsyncOperationResponse { + const ubyte[] bufferRead; + const SystemErrorCode errorCode; + + this(SystemErrorCode errorCode, const(ubyte)[] bufferRead) { + this.errorCode = errorCode; + this.bufferRead = bufferRead; + } + + override bool wasSuccessful() { + return errorCode.wasSuccessful; + } +} + +/+ + Tasks: + startTask() + startSubTask() - what if it just did this when it knows it is being run from inside a task? + runHelperFunction() - whomever it reports to is the parent ++/ + +/+ +class Task : Fiber { + +} ++/ + +private class CoreWorkerThread : Thread { + this(EventLoopType type) { + this.type = type; + + // task runners are supposed to have smallish stacks since they either just run a single callback or call into fibers + // the helper runners might be a bit bigger tho + super(&run); + } + void run() { + eventLoop = getThisThreadEventLoop(this.type); + atomicOp!"+="(startedCount, 1); + atomicOp!"+="(runningCount, 1); + scope(exit) { + atomicOp!"-="(runningCount, 1); + } + + eventLoop.run(() => true); + } + + EventLoopType type; + ICoreEventLoop eventLoop; + + __gshared static { + CoreWorkerThread[] taskRunners; + CoreWorkerThread[] helperRunners; + ICoreEventLoop mainThreadLoop; + + // for the helper function thing on the bsds i could have my own little circular buffer of availability + + shared(int) startedCount; + shared(int) runningCount; + + bool started; + + void setup(int numberOfTaskRunners, int numberOfHelpers) { + assert(!started); + synchronized { + mainThreadLoop = getThisThreadEventLoop(); + + foreach(i; 0 .. numberOfTaskRunners) { + auto nt = new CoreWorkerThread(EventLoopType.TaskRunner); + taskRunners ~= nt; + nt.start(); + } + foreach(i; 0 .. numberOfHelpers) { + auto nt = new CoreWorkerThread(EventLoopType.HelperWorker); + helperRunners ~= nt; + nt.start(); + } + + const expectedCount = numberOfHelpers + numberOfTaskRunners; + + while(startedCount < expectedCount) { + Thread.yield(); + } + + started = true; + } + } + } +} + +private int numberOfCpus() { + return 4; // FIXME +} + +/++ + To opt in to the full functionality of this module with customization opportunity, create one and only one of these objects that is valid for exactly the lifetime of the application. + + Normally, this means writing a main like this: + + --- + import arsd.core; + void main() { + ArsdCoreApplication app = ArsdCoreApplication("Your app name"); + + // do your setup here + + // the rest of your code here + } + --- + + Its destructor runs the event loop then waits to for the workers to finish to clean them up. ++/ +struct ArsdCoreApplication { + private ICoreEventLoop impl; + + /++ + default number of threads is to split your cpus between blocking function runners and task runners + +/ + this(string applicationName) { + auto num = numberOfCpus(); + num /= 2; + if(num <= 0) + num = 1; + this(applicationName, num, num); + } + + /++ + + +/ + this(string applicationName, int numberOfTaskRunners, int numberOfHelpers) { + impl = getThisThreadEventLoop(EventLoopType.Explicit); + CoreWorkerThread.setup(numberOfTaskRunners, numberOfHelpers); + } + + @disable this(); + @disable this(this); + /++ + This must be deterministically destroyed. + +/ + @disable new(); + + ~this() { + run(); + exitApplication(); + waitForWorkersToExit(3000); + } + + void exitApplication() { + + } + + void waitForWorkersToExit(int timeoutMilliseconds) { + + } + + void run() { + impl.run(() => true); + } +} + + +private class CoreEventLoopImplementation : ICoreEventLoop { + + version(Arsd_core_kqueue) { + // this thread apc dispatches go as a custom event to the queue + // the other queues go through one byte at a time pipes (barf). freebsd 13 and newest nbsd have eventfd too tho so maybe i can use them but the other kqueue systems don't. + + void runOnce() { + kevent_t[16] ev; + //timespec tout = timespec(1, 0); + auto nev = kevent(kqueuefd, null, 0, ev.ptr, ev.length, null/*&tout*/); + if(nev == -1) { + // FIXME: EINTR + throw new SystemApiException("kevent", errno); + } else if(nev == 0) { + // timeout + } else { + foreach(event; ev[0 .. nev]) { + if(event.filter == EVFILT_SIGNAL) { + // FIXME: I could prolly do this better tbh + markSignalOccurred(cast(int) event.ident); + signalChecker(); + } else { + // FIXME: event.filter more specific? + CallbackHelper cb = cast(CallbackHelper) event.udata; + cb.call(); + } + } + } + } + + // FIXME: idk how to make one event that multiple kqueues can listen to w/o being shared + // maybe a shared kqueue could work that the thread kqueue listen to (which i rejected for + // epoll cuz it caused thundering herd problems but maybe it'd work here) + + UnregisterToken addCallbackOnFdReadable(int fd, CallbackHelper cb) { + kevent_t ev; + + EV_SET(&ev, fd, EVFILT_READ, EV_ADD | EV_ENABLE/* | EV_ONESHOT*/, 0, 0, cast(void*) cb); + + ErrnoEnforce!kevent(kqueuefd, &ev, 1, null, 0, null); + + return UnregisterToken(this, fd, cb); + } + + RearmToken addCallbackOnFdReadableOneShot(int fd, CallbackHelper cb) { + kevent_t ev; + + EV_SET(&ev, fd, EVFILT_READ, EV_ADD | EV_ENABLE/* | EV_ONESHOT*/, 0, 0, cast(void*) cb); + + ErrnoEnforce!kevent(kqueuefd, &ev, 1, null, 0, null); + + return RearmToken(true, this, fd, cb, 0); + } + + RearmToken addCallbackOnFdWritableOneShot(int fd, CallbackHelper cb) { + kevent_t ev; + + EV_SET(&ev, fd, EVFILT_WRITE, EV_ADD | EV_ENABLE/* | EV_ONESHOT*/, 0, 0, cast(void*) cb); + + ErrnoEnforce!kevent(kqueuefd, &ev, 1, null, 0, null); + + return RearmToken(false, this, fd, cb, 0); + } + + private void rearmFd(RearmToken token) { + if(token.readable) + cast(void) addCallbackOnFdReadableOneShot(token.fd, token.cb); + else + cast(void) addCallbackOnFdWritableOneShot(token.fd, token.cb); + } + + private void triggerGlobalEvent() { + ubyte a; + import core.sys.posix.unistd; + write(kqueueGlobalFd[1], &a, 1); + } + + private this() { + kqueuefd = ErrnoEnforce!kqueue(); + setCloExec(kqueuefd); // FIXME O_CLOEXEC + + if(kqueueGlobalFd[0] == 0) { + import core.sys.posix.unistd; + pipe(kqueueGlobalFd); + setCloExec(kqueueGlobalFd[0]); + setCloExec(kqueueGlobalFd[1]); + + signal(SIGINT, SIG_IGN); // FIXME + } + + kevent_t ev; + + EV_SET(&ev, SIGCHLD, EVFILT_SIGNAL, EV_ADD | EV_ENABLE, 0, 0, null); + ErrnoEnforce!kevent(kqueuefd, &ev, 1, null, 0, null); + EV_SET(&ev, SIGINT, EVFILT_SIGNAL, EV_ADD | EV_ENABLE, 0, 0, null); + ErrnoEnforce!kevent(kqueuefd, &ev, 1, null, 0, null); + + globalEventSent = new CallbackHelper(&readGlobalEvent); + EV_SET(&ev, kqueueGlobalFd[0], EVFILT_READ, EV_ADD | EV_ENABLE, 0, 0, cast(void*) globalEventSent); + ErrnoEnforce!kevent(kqueuefd, &ev, 1, null, 0, null); + } + + private int kqueuefd = -1; + + private CallbackHelper globalEventSent; + void readGlobalEvent() { + kevent_t event; + + import core.sys.posix.unistd; + ubyte a; + read(kqueueGlobalFd[0], &a, 1); + + // FIXME: the thread is woken up, now we need to check the circualr buffer queue + } + + private __gshared int[2] kqueueGlobalFd; + } + + /+ + // this setup needs no extra allocation + auto op = read(file, buffer); + op.oncomplete = &thisfiber.call; + op.start(); + thisfiber.yield(); + auto result = op.waitForCompletion(); // guaranteed to return instantly thanks to previous setup + + can generically abstract that into: + + auto result = thisTask.await(read(file, buffer)); + + + You MUST NOT use buffer in any way - not read, modify, deallocate, reuse, anything - until the PendingOperation is complete. + + Note that PendingOperation may just be a wrapper around an internally allocated object reference... but then if you do a waitForFirstToComplete what happens? + + those could of course just take the value type things + +/ + + + version(Arsd_core_windows) { + // all event loops share the one iocp, Windows + // manages how to do it + __gshared HANDLE iocpTaskRunners; + __gshared HANDLE iocpWorkers; + + HANDLE[] handles; + + // i think to terminate i just have to post the message at least once for every thread i know about, maybe a few more times for threads i don't know about. + + bool isWorker; // if it is a worker we wait on the iocp, if not we wait on msg + + void runOnce() { + if(isWorker) { + // this function is only supported on Windows Vista and up, so using this + // means dropping support for XP. + //GetQueuedCompletionStatusEx(); + assert(0); // FIXME + } else { + auto wto = 0; + + auto waitResult = MsgWaitForMultipleObjectsEx( + cast(int) handles.length, handles.ptr, + (wto == 0 ? INFINITE : wto), /* timeout */ + 0x04FF, /* QS_ALLINPUT */ + 0x0002 /* MWMO_ALERTABLE */ | 0x0004 /* MWMO_INPUTAVAILABLE */); + + enum WAIT_OBJECT_0 = 0; + if(waitResult >= WAIT_OBJECT_0 && waitResult < handles.length + WAIT_OBJECT_0) { + auto h = handles[waitResult - WAIT_OBJECT_0]; + // FIXME: run the handle ready callback + } else if(waitResult == handles.length + WAIT_OBJECT_0) { + // message ready + int count; + MSG message; + while(PeekMessage(&message, null, 0, 0, PM_NOREMOVE)) { // need to peek since sometimes MsgWaitForMultipleObjectsEx returns even though GetMessage can block. tbh i don't fully understand it but the docs say it is foreground activation + auto ret = GetMessage(&message, null, 0, 0); + if(ret == -1) + throw new WindowsApiException("GetMessage", GetLastError()); + TranslateMessage(&message); + DispatchMessage(&message); + + count++; + if(count > 10) + break; // take the opportunity to catch up on other events + + if(ret == 0) { // WM_QUIT + // EventLoop.quitApplication(); + assert(0); // FIXME + //break; + } + } + } else if(waitResult == 0x000000C0L /* WAIT_IO_COMPLETION */) { + SleepEx(0, true); // I call this to give it a chance to do stuff like async io + } else if(waitResult == 258L /* WAIT_TIMEOUT */) { + // timeout, should never happen since we aren't using it + } else if(waitResult == 0xFFFFFFFF) { + // failed + throw new WindowsApiException("MsgWaitForMultipleObjectsEx", GetLastError()); + } else { + // idk.... + } + } + } + } + + version(Posix) { + private __gshared uint sigChildHappened = 0; + private __gshared uint sigIntrHappened = 0; + + static void signalChecker() { + if(cas(&sigChildHappened, 1, 0)) { + while(true) { // multiple children could have exited before we processed the notification + + import core.sys.posix.sys.wait; + + int status; + auto pid = waitpid(-1, &status, WNOHANG); + if(pid == -1) { + import core.stdc.errno; + auto errno = errno; + if(errno == ECHILD) + break; // also all done, there are no children left + // no need to check EINTR since we set WNOHANG + throw new ErrnoApiException("waitpid", errno); + } + if(pid == 0) + break; // all done, all children are still running + + // look up the pid for one of our objects + // if it is found, inform it of its status + // and then inform its controlling thread + // to wake up so it can check its waitForCompletion, + // trigger its callbacks, etc. + + ExternalProcess.recordChildTerminated(pid, status); + } + + } + if(cas(&sigIntrHappened, 1, 0)) { + // FIXME + import core.stdc.stdlib; + exit(0); + } + } + + /++ + Informs the arsd.core system that the given signal happened. You can call this from inside a signal handler. + +/ + public static void markSignalOccurred(int sigNumber) nothrow { + import core.sys.posix.unistd; + + if(sigNumber == SIGCHLD) + volatileStore(&sigChildHappened, 1); + if(sigNumber == SIGINT) + volatileStore(&sigIntrHappened, 1); + + version(Arsd_core_epoll) { + ulong writeValue = 1; + write(signalPipeFd, &writeValue, writeValue.sizeof); + } + } + } + + version(Arsd_core_epoll) { + + import core.sys.linux.epoll; + import core.sys.linux.sys.eventfd; + + private this() { + + if(!globalsInitialized) { + synchronized { + if(!globalsInitialized) { + // blocking signals is problematic because it is inherited by child processes + // and that can be problematic for general purpose stuff so i use a self pipe + // here. though since it is linux, im using an eventfd instead just to notify + signalPipeFd = ErrnoEnforce!eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK); + signalReaderCallback = new CallbackHelper(&signalReader); + + runInTaskRunnerQueue = new CallbackQueue("task runners", true); + runInHelperThreadQueue = new CallbackQueue("helper threads", true); + + setSignalHandlers(); + + globalsInitialized = true; + } + } + } + + epollfd = epoll_create1(EPOLL_CLOEXEC); + + // FIXME: ensure UI events get top priority + + // global listeners + + // FIXME: i should prolly keep the tokens and release them when tearing down. + + cast(void) addCallbackOnFdReadable(signalPipeFd, signalReaderCallback); + if(true) { // FIXME: if this is a task runner vs helper thread vs ui thread + cast(void) addCallbackOnFdReadable(runInTaskRunnerQueue.fd, runInTaskRunnerQueue.callback); + runInTaskRunnerQueue.callback.addref(); + } else { + cast(void) addCallbackOnFdReadable(runInHelperThreadQueue.fd, runInHelperThreadQueue.callback); + runInHelperThreadQueue.callback.addref(); + } + + // local listener + thisThreadQueue = new CallbackQueue("this thread", false); + cast(void) addCallbackOnFdReadable(thisThreadQueue.fd, thisThreadQueue.callback); + + // what are we going to do about timers? + } + + void teardown() { + import core.sys.posix.fcntl; + import core.sys.posix.unistd; + + close(epollfd); + epollfd = -1; + + thisThreadQueue.teardown(); + + // FIXME: should prolly free anything left in the callback queue, tho those could also be GC managed tbh. + } + + /+ // i call it explicitly at the thread exit instead, but worker threads aren't really supposed to exit generally speaking till process done anyway + static ~this() { + teardown(); + } + +/ + + static void teardownGlobals() { + import core.sys.posix.fcntl; + import core.sys.posix.unistd; + + synchronized { + restoreSignalHandlers(); + close(signalPipeFd); + signalReaderCallback.release(); + + runInTaskRunnerQueue.teardown(); + runInHelperThreadQueue.teardown(); + + globalsInitialized = false; + } + + } + + + private static final class CallbackQueue { + int fd = -1; + string name; + CallbackHelper callback; + SynchronizedCircularBuffer!CallbackHelper queue; + + this(string name, bool dequeueIsShared) { + this.name = name; + queue = typeof(queue)(this); + + fd = ErrnoEnforce!eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK | (dequeueIsShared ? EFD_SEMAPHORE : 0)); + + callback = new CallbackHelper(dequeueIsShared ? &sharedDequeueCb : &threadLocalDequeueCb); + } + + bool resetEvent() { + import core.sys.posix.unistd; + ulong count; + return read(fd, &count, count.sizeof) == count.sizeof; + } + + void sharedDequeueCb() { + if(resetEvent()) { + auto cb = queue.dequeue(); + cb.call(); + cb.release(); + } + } + + void threadLocalDequeueCb() { + CallbackHelper[16] buffer; + foreach(cb; queue.dequeueSeveral(buffer[], () { resetEvent(); })) { + cb.call(); + cb.release(); + } + } + + void enqueue(CallbackHelper cb) { + if(queue.enqueue(cb)) { + import core.sys.posix.unistd; + ulong count = 1; + ErrnoEnforce!write(fd, &count, count.sizeof); + } else { + throw new ArsdException!"queue is full"(name); + } + } + + void teardown() { + import core.sys.posix.fcntl; + import core.sys.posix.unistd; + + close(fd); + fd = -1; + + callback.release(); + } + } + + // there's a global instance of this we refer back to + private __gshared { + bool globalsInitialized; + + CallbackHelper signalReaderCallback; + + CallbackQueue runInTaskRunnerQueue; + CallbackQueue runInHelperThreadQueue; + + int exitEventFd = -1; // FIXME: implement + } + + // and then the local loop + private { + int epollfd = -1; + + CallbackQueue thisThreadQueue; + } + + // signal stuff { + import core.sys.posix.signal; + + private __gshared sigaction_t oldSigIntr; + private __gshared sigaction_t oldSigChld; + private __gshared sigaction_t oldSigPipe; + + private __gshared int signalPipeFd = -1; + // sigpipe not important, i handle errors on the writes + + public static void setSignalHandlers() { + static extern(C) void interruptHandler(int sigNumber) nothrow { + markSignalOccurred(sigNumber); + + /+ + // calling the old handler is non-trivial since there can be ignore + // or default or a plain handler or a sigaction 3 arg handler and i + // i don't think it is worth teh complication + sigaction_t* oldHandler; + if(sigNumber == SIGCHLD) + oldHandler = &oldSigChld; + else if(sigNumber == SIGINT) + oldHandler = &oldSigIntr; + if(oldHandler && oldHandler.sa_handler) + oldHandler + +/ + } + + sigaction_t n; + n.sa_handler = &interruptHandler; + n.sa_mask = cast(sigset_t) 0; + n.sa_flags = 0; + sigaction(SIGINT, &n, &oldSigIntr); + sigaction(SIGCHLD, &n, &oldSigChld); + + n.sa_handler = SIG_IGN; + sigaction(SIGPIPE, &n, &oldSigPipe); + } + + public static void restoreSignalHandlers() { + sigaction(SIGINT, &oldSigIntr, null); + sigaction(SIGCHLD, &oldSigChld, null); + sigaction(SIGPIPE, &oldSigPipe, null); + } + + private static void signalReader() { + import core.sys.posix.unistd; + ulong number; + read(signalPipeFd, &number, number.sizeof); + + signalChecker(); + } + // signal stuff done } + + // the any thread poll is just registered in the this thread poll w/ exclusive. nobody actaully epoll_waits + // on the global one directly. + + void runOnce() { + epoll_event[16] events; + auto ret = epoll_wait(epollfd, events.ptr, cast(int) events.length, -1); // FIXME: timeout + if(ret == -1) { + import core.stdc.errno; + if(errno == EINTR) { + return; + } + throw new ErrnoApiException("epoll_wait", errno); + } else if(ret == 0) { + // timeout + } else { + // loop events and call associated callbacks + foreach(event; events[0 .. ret]) { + auto flags = event.events; + auto cbObject = cast(CallbackHelper) event.data.ptr; + + // FIXME: or if it is an error... + // EPOLLERR - write end of pipe when read end closed or other error. and EPOLLHUP - terminal hangup or read end when write end close (but it will give 0 reading after that soon anyway) + + cbObject.call(); + } + } + } + + // building blocks for low-level integration with the loop + + UnregisterToken addCallbackOnFdReadable(int fd, CallbackHelper cb) { + epoll_event event; + event.data.ptr = cast(void*) cb; + event.events = EPOLLIN | EPOLLEXCLUSIVE; + if(epoll_ctl(epollfd, EPOLL_CTL_ADD, fd, &event) == -1) + throw new ErrnoApiException("epoll_ctl", errno); + + return UnregisterToken(this, fd, cb); + } + + /++ + Adds a one-off callback that you can optionally rearm when it happens. + +/ + RearmToken addCallbackOnFdReadableOneShot(int fd, CallbackHelper cb) { + epoll_event event; + event.data.ptr = cast(void*) cb; + event.events = EPOLLIN | EPOLLONESHOT; + if(epoll_ctl(epollfd, EPOLL_CTL_ADD, fd, &event) == -1) + throw new ErrnoApiException("epoll_ctl", errno); + + return RearmToken(true, this, fd, cb, EPOLLIN | EPOLLONESHOT); + } + + /++ + Adds a one-off callback that you can optionally rearm when it happens. + +/ + RearmToken addCallbackOnFdWritableOneShot(int fd, CallbackHelper cb) { + epoll_event event; + event.data.ptr = cast(void*) cb; + event.events = EPOLLOUT | EPOLLONESHOT; + if(epoll_ctl(epollfd, EPOLL_CTL_ADD, fd, &event) == -1) + throw new ErrnoApiException("epoll_ctl", errno); + + return RearmToken(false, this, fd, cb, EPOLLOUT | EPOLLONESHOT); + } + + private void unregisterFd(int fd) { + epoll_event event; + if(epoll_ctl(epollfd, EPOLL_CTL_DEL, fd, &event) == -1) + throw new ErrnoApiException("epoll_ctl", errno); + } + + private void rearmFd(RearmToken token) { + epoll_event event; + event.data.ptr = cast(void*) token.cb; + event.events = token.flags; + if(epoll_ctl(epollfd, EPOLL_CTL_MOD, token.fd, &event) == -1) + throw new ErrnoApiException("epoll_ctl", errno); + } + + // Disk files will have to be sent as messages to a worker to do the read and report back a completion packet. + } + + version(Arsd_core_kqueue) { + // FIXME + } + + // cross platform adapters + void setTimeout() {} + void addFileOrDirectoryChangeListener(FilePath name, uint flags, bool recursive = false) {} +} + +// deduplication???????// +bool postMessage(ThreadToRunIn destination, void delegate() code) { + return false; +} +bool postMessage(ThreadToRunIn destination, Object message) { + return false; +} + +/+ +void main() { + // FIXME: the offset doesn't seem to be done right + auto file = new AsyncFile(FilePath("test.txt"), AsyncFile.OpenMode.writeWithTruncation); + file.write("hello", 10).waitForCompletion(); +} ++/ + +// to test the mailboxes +/+ +void main() { + /+ + import std.stdio; + Thread[4] pool; + + bool shouldExit; + + static int received; + + static void tester() { + received++; + //writeln(cast(void*) Thread.getThis, " ", received); + } + + foreach(ref thread; pool) { + thread = new Thread(() { + getThisThreadEventLoop().run(() { + return shouldExit; + }); + }); + thread.start(); + } + + getThisThreadEventLoop(); // ensure it is all initialized before proceeding. FIXME: i should have an ensure initialized function i do on most the public apis. + + int lol; + + try + foreach(i; 0 .. 6000) { + CoreEventLoopImplementation.runInTaskRunnerQueue.enqueue(new CallbackHelper(&tester)); + lol = cast(int) i; + } + catch(ArsdExceptionBase e) { + Thread.sleep(50.msecs); + writeln(e); + writeln(lol); + } + + import core.stdc.stdlib; + exit(0); + + version(none) + foreach(i; 0 .. 100) + CoreEventLoopImplementation.runInTaskRunnerQueue.enqueue(new CallbackHelper(&tester)); + + + foreach(ref thread; pool) { + thread.join(); + } + +/ + + + static int received; + + static void tester() { + received++; + //writeln(cast(void*) Thread.getThis, " ", received); + } + + + + auto ev = cast(CoreEventLoopImplementation) getThisThreadEventLoop(); + foreach(i; 0 .. 100) + ev.thisThreadQueue.enqueue(new CallbackHelper(&tester)); + foreach(i; 0 .. 100 / 16 + 1) + ev.runOnce(); + import std.conv; + assert(received == 100, to!string(received)); + +} ++/ + +/++ + This is primarily a helper for the event queues. It is public in the hope it might be useful, + but subject to change without notice; I will treat breaking it the same as if it is private. + (That said, it is a simple little utility that does its job, so it is unlikely to change much. + The biggest change would probably be letting it grow and changing from inline to dynamic array.) + + It is a fixed-size ring buffer that synchronizes on a given object you give it in the constructor. + + After enqueuing something, you should probably set an event to notify the other threads. This is left + as an exercise to you (or another wrapper). ++/ +struct SynchronizedCircularBuffer(T, size_t maxSize = 128) { + private T[maxSize] ring; + private int front; + private int back; + + private Object synchronizedOn; + + @disable this(); + + /++ + The Object's monitor is used to synchronize the methods in here. + +/ + this(Object synchronizedOn) { + this.synchronizedOn = synchronizedOn; + } + + /++ + Note the potential race condition between calling this and actually dequeuing something. You might + want to acquire the lock on the object before calling this (nested synchronized things are allowed + as long as the same thread is the one doing it). + +/ + bool isEmpty() { + synchronized(this.synchronizedOn) { + return front == back; + } + } + + /++ + Note the potential race condition between calling this and actually queuing something. + +/ + bool isFull() { + synchronized(this.synchronizedOn) { + return isFullUnsynchronized(); + } + } + + private bool isFullUnsynchronized() nothrow const { + return ((back + 1) % ring.length) == front; + + } + + /++ + If this returns true, you should signal listening threads (with an event or a semaphore, + depending on how you dequeue it). If it returns false, the queue was full and your thing + was NOT added. You might wait and retry later (you could set up another event to signal it + has been read and wait for that, or maybe try on a timer), or just fail and throw an exception + or to abandon the message. + +/ + bool enqueue(T what) { + synchronized(this.synchronizedOn) { + if(isFullUnsynchronized()) + return false; + ring[(back++) % ring.length] = what; + return true; + } + } + + private T dequeueUnsynchronized() nothrow { + assert(front != back); + return ring[(front++) % ring.length]; + } + + /++ + If you are using a semaphore to signal, you can call this once for each count of it + and you can do that separately from this call (though they should be paired). + + If you are using an event, you should use [dequeueSeveral] instead to drain it. + +/ + T dequeue() { + synchronized(this.synchronizedOn) { + return dequeueUnsynchronized(); + } + } + + /++ + Note that if you use a semaphore to signal waiting threads, you should probably not call this. + + If you use a set/reset event, there's a potential race condition between the dequeue and event + reset. This is why the `runInsideLockIfEmpty` delegate is there - when it is empty, before it + unlocks, it will give you a chance to reset the event. Otherwise, it can remain set to indicate + that there's still pending data in the queue. + +/ + T[] dequeueSeveral(return T[] buffer, scope void delegate() runInsideLockIfEmpty = null) { + int pos; + synchronized(this.synchronizedOn) { + while(pos < buffer.length && front != back) { + buffer[pos++] = dequeueUnsynchronized(); + } + if(front == back && runInsideLockIfEmpty !is null) + runInsideLockIfEmpty(); + } + return buffer[0 .. pos]; + } +} + +unittest { + Object object = new Object(); + auto queue = SynchronizedCircularBuffer!CallbackHelper(object); + assert(queue.isEmpty); + foreach(i; 0 .. queue.ring.length - 1) + queue.enqueue(cast(CallbackHelper) cast(void*) i); + assert(queue.isFull); + + foreach(i; 0 .. queue.ring.length - 1) + assert(queue.dequeue() is (cast(CallbackHelper) cast(void*) i)); + assert(queue.isEmpty); + + foreach(i; 0 .. queue.ring.length - 1) + queue.enqueue(cast(CallbackHelper) cast(void*) i); + assert(queue.isFull); + + CallbackHelper[] buffer = new CallbackHelper[](300); + auto got = queue.dequeueSeveral(buffer); + assert(got.length == queue.ring.length - 1); + assert(queue.isEmpty); + foreach(i, item; got) + assert(item is (cast(CallbackHelper) cast(void*) i)); + + foreach(i; 0 .. 8) + queue.enqueue(cast(CallbackHelper) cast(void*) i); + buffer = new CallbackHelper[](4); + got = queue.dequeueSeveral(buffer); + assert(got.length == 4); + foreach(i, item; got) + assert(item is (cast(CallbackHelper) cast(void*) i)); + got = queue.dequeueSeveral(buffer); + assert(got.length == 4); + foreach(i, item; got) + assert(item is (cast(CallbackHelper) cast(void*) (i+4))); + got = queue.dequeueSeveral(buffer); + assert(got.length == 0); + assert(queue.isEmpty); +} + +/++ + ++/ +enum ByteOrder { + irrelevant, + littleEndian, + bigEndian, +} + +/++ + A class to help write a stream of binary data to some target. + + NOT YET FUNCTIONAL ++/ +class WritableStream { + /++ + + +/ + this(size_t bufferSize) { + this(new ubyte[](bufferSize)); + } + + /// ditto + this(ubyte[] buffer) { + this.buffer = buffer; + } + + /++ + + +/ + final void put(T)(T value, ByteOrder byteOrder = ByteOrder.irrelevant) { + static if(T.sizeof == 8) + ulong b; + else static if(T.sizeof == 4) + uint b; + else static if(T.sizeof == 2) + ushort b; + else static if(T.sizeof == 1) + ubyte b; + else static assert(0, "unimplemented type, try using just the basic types"); + + if(byteOrder == ByteOrder.irrelevant && T.sizeof > 1) + throw new InvalidArgumentsException("byteOrder", "byte order must be specified for type " ~ T.stringof ~ " because it is bigger than one byte"); + + final switch(byteOrder) { + case ByteOrder.irrelevant: + writeOneByte(b); + break; + case ByteOrder.littleEndian: + foreach(i; 0 .. T.sizeof) { + writeOneByte(b & 0xff); + b >>= 8; + } + break; + case ByteOrder.bigEndian: + int amount = T.sizeof * 8 - 8; + foreach(i; 0 .. T.sizeof) { + writeOneByte((b >> amount) & 0xff); + amount -= 8; + } + break; + } + } + + /// ditto + final void put(T : E[], E)(T value, ByteOrder elementByteOrder = ByteOrder.irrelevant) { + foreach(item; value) + put(item, elementByteOrder); + } + + /++ + Performs a final flush() call, then marks the stream as closed, meaning no further data will be written to it. + +/ + void close() { + isClosed_ = true; + } + + /++ + Writes what is currently in the buffer to the target and waits for the target to accept it. + Please note: if you are subclassing this to go to a different target + +/ + void flush() {} + + /++ + Returns true if either you closed it or if the receiving end closed their side, indicating they + don't want any more data. + +/ + bool isClosed() { + return isClosed_; + } + + // hasRoomInBuffer + // canFlush + // waitUntilCanFlush + + // flushImpl + // markFinished / close - tells the other end you're done + + private final writeOneByte(ubyte value) { + if(bufferPosition == buffer.length) + flush(); + + buffer[bufferPosition++] = value; + } + + + private { + ubyte[] buffer; + int bufferPosition; + bool isClosed_; + } +} + +/++ + A stream can be used by just one task at a time, but one task can consume multiple streams. + + Streams may be populated by async sources (in which case they must be called from a fiber task), + from a function generating the data on demand (including an input range), from memory, or from a synchronous file. + + A stream of heterogeneous types is compatible with input ranges. + + It reads binary data. ++/ +class ReadableStream { + + this() { + + } + + /++ + Gets data of the specified type `T` off the stream. The byte order of the T on the stream must be specified unless it is irrelevant (e.g. single byte entries). + + --- + // get an int out of a big endian stream + int i = stream.get!int(ByteOrder.bigEndian); + + // get i bytes off the stream + ubyte[] data = stream.get!(ubyte[])(i); + --- + +/ + final T get(T)(ByteOrder byteOrder = ByteOrder.irrelevant) { + if(byteOrder == ByteOrder.irrelevant && T.sizeof > 1) + throw new InvalidArgumentsException("byteOrder", "byte order must be specified for type " ~ T.stringof ~ " because it is bigger than one byte"); + + // FIXME: what if it is a struct? + + while(bufferedLength() < T.sizeof) + waitForAdditionalData(); + + static if(T.sizeof == 1) { + ubyte ret = consumeOneByte(); + return *cast(T*) &ret; + } else { + static if(T.sizeof == 8) + ulong ret; + else static if(T.sizeof == 4) + uint ret; + else static if(T.sizeof == 2) + ushort ret; + else static assert(0, "unimplemented type, try using just the basic types"); + + if(byteOrder == ByteOrder.littleEndian) { + typeof(ret) buffer; + foreach(b; 0 .. T.sizeof) { + buffer = consumeOneByte(); + buffer <<= T.sizeof * 8 - 8; + + ret >>= 8; + ret |= buffer; + } + } else { + foreach(b; 0 .. T.sizeof) { + ret <<= 8; + ret |= consumeOneByte(); + } + } + + return *cast(T*) &ret; + } + } + + /// ditto + final T get(T : E[], E)(size_t length, ByteOrder elementByteOrder = ByteOrder.irrelevant) { + if(elementByteOrder == ByteOrder.irrelevant && E.sizeof > 1) + throw new InvalidArgumentsException("elementByteOrder", "byte order must be specified for type " ~ E.stringof ~ " because it is bigger than one byte"); + + // if the stream is closed before getting the length or the terminator, should we send partial stuff + // or just throw? + + while(bufferedLength() < length * E.sizeof) + waitForAdditionalData(); + + T ret; + + ret.length = length; + + if(false && elementByteOrder == ByteOrder.irrelevant) { + // ret[] = + // FIXME: can prolly optimize + } else { + foreach(i; 0 .. length) + ret[i] = get!E(elementByteOrder); + } + + return ret; + + } + + /// ditto + final T get(T : E[], E)(scope bool delegate(E e) isTerminatingSentinel, ByteOrder elementByteOrder = ByteOrder.irrelevant) { + if(byteOrder == ByteOrder.irrelevant && E.sizeof > 1) + throw new InvalidArgumentsException("elementByteOrder", "byte order must be specified for type " ~ E.stringof ~ " because it is bigger than one byte"); + + assert(0, "Not implemented"); + } + + /++ + + +/ + bool isClosed() { + return isClosed_; + } + + // Control side of things + + private bool isClosed_; + + /++ + Feeds data into the stream, which can be consumed by `get`. If a task is waiting for more + data to satisfy its get requests, this will trigger those tasks to resume. + + If you feed it empty data, it will mark the stream as closed. + +/ + void feedData(ubyte[] data) { + if(data.length == 0) + isClosed_ = true; + + currentBuffer = data; + // this is a borrowed buffer, so we won't keep the reference long term + scope(exit) + currentBuffer = null; + + if(waitingTask !is null) { + waitingTask.call(); + } + } + + /++ + You basically have to use this thing from a task + +/ + protected void waitForAdditionalData() { + Fiber task = Fiber.getThis; + + assert(task !is null); + + if(waitingTask !is null && waitingTask !is task) + throw new ArsdException!"streams can only have one waiting task"; + + // copy any pending data in our buffer to the longer-term buffer + if(currentBuffer.length) + leftoverBuffer ~= currentBuffer; + + waitingTask = task; + task.yield(); + } + + private Fiber waitingTask; + private ubyte[] leftoverBuffer; + private ubyte[] currentBuffer; + + private size_t bufferedLength() { + return leftoverBuffer.length + currentBuffer.length; + } + + private ubyte consumeOneByte() { + ubyte b; + if(leftoverBuffer.length) { + b = leftoverBuffer[0]; + leftoverBuffer = leftoverBuffer[1 .. $]; + } else if(currentBuffer.length) { + b = currentBuffer[0]; + currentBuffer = currentBuffer[1 .. $]; + } else { + assert(0, "consuming off an empty buffer is impossible"); + } + + return b; + } +} + +// FIXME: do a stringstream too + +unittest { + auto stream = new ReadableStream(); + + int position; + char[16] errorBuffer; + + auto fiber = new Fiber(() { + position = 1; + int a = stream.get!int(ByteOrder.littleEndian); + assert(a == 10, intToString(a, errorBuffer[])); + position = 2; + ubyte b = stream.get!ubyte; + assert(b == 33); + position = 3; + + // ubyte[] c = stream.get!(ubyte[])(3); + // int[] d = stream.get!(int[])(3); + }); + + fiber.call(); + assert(position == 1); + stream.feedData([10, 0, 0, 0]); + assert(position == 2); + stream.feedData([33]); + assert(position == 3); + + // stream.feedData([1,2,3]); + // stream.feedData([1,2,3,4,1,2,3,4,1,2,3,4]); +} + +/++ + You might use this like: + + --- + auto proc = new ExternalProcess(); + auto stdoutStream = new ReadableStream(); + + // to use a stream you can make one and have a task consume it + runTask({ + while(!stdoutStream.isClosed) { + auto line = stdoutStream.get!string(e => e == '\n'); + } + }); + + // then make the process feed into the stream + proc.onStdoutAvailable = (got) { + stdoutStream.feedData(got); // send it to the stream for processing + stdout.rawWrite(got); // forward it through to our own thing + // could also append it to a buffer to return it on complete + }; + proc.start(); + --- + + Please note that this does not currently and I have no plans as of this writing to add support for any kind of direct file descriptor passing. It always pipes them back to the parent for processing. If you don't want this, call the lower level functions yourself; the reason this class is here is to aid integration in the arsd.core event loop. + + Of course, I might change my mind on this. ++/ +class ExternalProcess { + + private static version(Posix) { + __gshared ExternalProcess[pid_t] activeChildren; + + void recordChildCreated(pid_t pid, ExternalProcess proc) { + synchronized(typeid(ExternalProcess)) { + activeChildren[pid] = proc; + } + } + + void recordChildTerminated(pid_t pid, int status) { + synchronized(typeid(ExternalProcess)) { + if(pid in activeChildren) { + auto ac = activeChildren[pid]; + ac.completed = true; + ac.status = status; + activeChildren.remove(pid); + } + } + } + } + + // FIXME: config to pass through a shell or not + + /++ + This is the native version for Windows. + +/ + this(string program, string commandLine) { + version(Posix) { + assert(0, "not implemented command line to posix args yet"); + } + } + + this(string commandLine) { + version(Posix) { + assert(0, "not implemented command line to posix args yet"); + } + } + + this(string[] args) { + version(Posix) { + this.program = FilePath(args[0]); + this.args = args; + } + + } + + /++ + This is the native version for Posix. + +/ + this(FilePath program, string[] args) { + version(Posix) { + this.program = program; + this.args = args; + } + } + + // you can modify these before calling start + int stdoutBufferSize = 32 * 1024; + int stderrBufferSize = 8 * 1024; + + void start() { + version(Posix) { + int ret; + + int[2] stdinPipes; + ret = pipe(stdinPipes); + if(ret == -1) + throw new ErrnoApiException("stdin pipe", errno); + + scope(failure) { + close(stdinPipes[0]); + close(stdinPipes[1]); + } + + stdinFd = stdinPipes[1]; + + int[2] stdoutPipes; + ret = pipe(stdoutPipes); + if(ret == -1) + throw new ErrnoApiException("stdout pipe", errno); + + scope(failure) { + close(stdoutPipes[0]); + close(stdoutPipes[1]); + } + + stdoutFd = stdoutPipes[0]; + + int[2] stderrPipes; + ret = pipe(stderrPipes); + if(ret == -1) + throw new ErrnoApiException("stderr pipe", errno); + + scope(failure) { + close(stderrPipes[0]); + close(stderrPipes[1]); + } + + stderrFd = stderrPipes[0]; + + + int[2] errorReportPipes; + ret = pipe(errorReportPipes); + if(ret == -1) + throw new ErrnoApiException("error reporting pipe", errno); + + scope(failure) { + close(errorReportPipes[0]); + close(errorReportPipes[1]); + } + + setCloExec(errorReportPipes[0]); + setCloExec(errorReportPipes[1]); + + auto forkRet = fork(); + if(forkRet == -1) + throw new ErrnoApiException("fork", errno); + + if(forkRet == 0) { + // child side + + // FIXME can we do more error checking that is actually useful here? + // these operations are virtually guaranteed to succeed given the setup anyway. + + // FIXME pty too + + void fail(int step) { + import core.stdc.errno; + auto code = errno; + + // report the info back to the parent then exit + + int[2] msg = [step, code]; + auto ret = write(errorReportPipes[1], msg.ptr, msg.sizeof); + + // but if this fails there's not much we can do... + + import core.stdc.stdlib; + exit(1); + } + + // dup2 closes the fd it is replacing automatically + dup2(stdinPipes[0], 0); + dup2(stdoutPipes[1], 1); + dup2(stderrPipes[1], 2); + + // don't need either of the original pipe fds anymore + close(stdinPipes[0]); + close(stdinPipes[1]); + close(stdoutPipes[0]); + close(stdoutPipes[1]); + close(stderrPipes[0]); + close(stderrPipes[1]); + + // the error reporting pipe will be closed upon exec since we set cloexec before fork + // and everything else should have cloexec set too hopefully. + + if(beforeExec) + beforeExec(); + + // i'm not sure that a fully-initialized druntime is still usable + // after a fork(), so i'm gonna stick to the C lib in here. + + const(char)* file = mallocedStringz(program.path).ptr; + if(file is null) + fail(1); + const(char)*[] argv = mallocSlice!(const(char)*)(args.length + 1); + if(argv is null) + fail(2); + foreach(idx, arg; args) { + argv[idx] = mallocedStringz(args[idx]).ptr; + if(argv[idx] is null) + fail(3); + } + argv[args.length] = null; + + auto rete = execvp/*e*/(file, argv.ptr/*, envp*/); + if(rete == -1) { + fail(4); + } else { + // unreachable code, exec never returns if it succeeds + assert(0); + } + } else { + pid = forkRet; + + recordChildCreated(pid, this); + + // close our copy of the write side of the error reporting pipe + // so the read will immediately give eof when the fork closes it too + ErrnoEnforce!close(errorReportPipes[1]); + + int[2] msg; + // this will block to wait for it to actually either start up or fail to exec (which should be near instant) + auto val = read(errorReportPipes[0], msg.ptr, msg.sizeof); + + if(val == -1) + throw new ErrnoApiException("read error report", errno); + + if(val == msg.sizeof) { + // error happened + // FIXME: keep the step part of the error report too + throw new ErrnoApiException("exec", msg[1]); + } else if(val == 0) { + // pipe closed, meaning exec succeeded + } else { + assert(0); // never supposed to happen + } + + // set the ones we keep to close upon future execs + // FIXME should i set NOBLOCK at this time too? prolly should + setCloExec(stdinPipes[1]); + setCloExec(stdoutPipes[0]); + setCloExec(stderrPipes[0]); + + // and close the others + ErrnoEnforce!close(stdinPipes[0]); + ErrnoEnforce!close(stdoutPipes[1]); + ErrnoEnforce!close(stderrPipes[1]); + + ErrnoEnforce!close(errorReportPipes[0]); + + // and now register the ones we need to read with the event loop so it can call the callbacks + // also need to listen to SIGCHLD to queue up the terminated callback. FIXME + + stdoutUnregisterToken = getThisThreadEventLoop().addCallbackOnFdReadable(stdoutFd, new CallbackHelper(&stdoutReadable)); + } + } + } + + private version(Posix) { + import core.sys.posix.unistd; + import core.sys.posix.fcntl; + + int stdinFd = -1; + int stdoutFd = -1; + int stderrFd = -1; + + ICoreEventLoop.UnregisterToken stdoutUnregisterToken; + + pid_t pid = -1; + + public void delegate() beforeExec; + + FilePath program; + string[] args; + + void stdoutReadable() { + ubyte[1024] buffer; + auto ret = read(stdoutFd, buffer.ptr, buffer.length); + if(ret == -1) + throw new ErrnoApiException("read", errno); + if(onStdoutAvailable) { + onStdoutAvailable(buffer[0 .. ret]); + } + + if(ret == 0) { + stdoutUnregisterToken.unregister(); + + close(stdoutFd); + stdoutFd = -1; + } + } + } + + void waitForCompletion() { + getThisThreadEventLoop().run(&this.isComplete); + } + + bool isComplete() { + return completed; + } + + bool completed; + int status = int.min; + + /++ + If blocking, it will block the current task until the write succeeds. + + Write `null` as data to close the pipe. Once the pipe is closed, you must not try to write to it again. + +/ + void writeToStdin(in void[] data) { + version(Posix) { + if(data is null) { + close(stdinFd); + stdinFd = -1; + } else { + // FIXME: check the return value again and queue async writes + auto ret = write(stdinFd, data.ptr, data.length); + if(ret == -1) + throw new ErrnoApiException("write", errno); + } + } + + } + + void delegate(ubyte[] got) onStdoutAvailable; + void delegate(ubyte[] got) onStderrAvailable; + void delegate(int code) onTermination; + + // pty? +} + +// FIXME: comment this out +/+ +unittest { + auto proc = new ExternalProcess(FilePath("/bin/cat"), ["/bin/cat"]); + + getThisThreadEventLoop(); // initialize it + + int c = 0; + proc.onStdoutAvailable = delegate(ubyte[] got) { + if(c == 0) + assert(cast(string) got == "hello!"); + else + assert(got.length == 0); + // import std.stdio; writeln(got); + c++; + }; + + proc.start(); + + assert(proc.pid != -1); + + + import std.stdio; + Thread[4] pool; + + bool shouldExit; + + static int received; + + static void tester() { + received++; + //writeln(cast(void*) Thread.getThis, " ", received); + } + + foreach(ref thread; pool) { + thread = new Thread(() { + getThisThreadEventLoop().run(() { + return shouldExit; + }); + }); + thread.start(); + } + + + + proc.writeToStdin("hello!"); + proc.writeToStdin(null); // closes the pipe + + proc.waitForCompletion(); + + assert(proc.status == 0); + + assert(c == 2); +} ++/ + +// to test the thundering herd on signal handling +version(none) +unittest { + Thread[4] pool; + foreach(ref thread; pool) { + thread = new class Thread { + this() { + super({ + int count; + getThisThreadEventLoop().run(() { + if(count > 4) return true; + count++; + return false; + }); + }); + } + }; + thread.start(); + } + foreach(ref thread; pool) { + thread.join(); + } +} + +/+ + ================= + STDIO REPLACEMENT + ================= ++/ + +/++ + A `writeln` that actually works. + + It works correctly on Windows, using the correct functions to write unicode to the console. even allocating a console if needed. If the output has been redirected to a file or pipe, it writes UTF-8. + + This always does text. See also WritableStream and WritableTextStream ++/ +void writeln(T...)(T t) { + char[256] bufferBacking; + char[] buffer = bufferBacking[]; + int pos; + foreach(arg; t) { + static if(is(typeof(arg) : const char[])) { + buffer[pos .. pos + arg.length] = arg[]; + pos += arg.length; + } else static if(is(typeof(arg) : stringz)) { + auto b = arg.borrow; + buffer[pos .. pos + b.length] = b[]; + pos += b.length; + } else static if(is(typeof(arg) : long)) { + auto sliced = intToString(arg, buffer[pos .. $]); + pos += sliced.length; + } else static if(is(typeof(arg.toString()) : const char[])) { + auto s = arg.toString(); + buffer[pos .. pos + s.length] = s[]; + pos += s.length; + } else { + auto s = "<unsupported type>"; + buffer[pos .. pos + s.length] = s[]; + pos += s.length; + // static assert(0, "Unsupported type: " ~ T.stringof); + } + } + + buffer[pos++] = '\n'; + + version(Windows) { + import core.sys.windows.wincon; + + auto hStdOut = GetStdHandle(STD_OUTPUT_HANDLE); + if(hStdOut == null || hStdOut == INVALID_HANDLE_VALUE) { + AllocConsole(); + hStdOut = GetStdHandle(STD_OUTPUT_HANDLE); + } + + if(GetFileType(hStdOut) == FILE_TYPE_CHAR) { + wchar[256] wbuffer; + auto toWrite = makeWindowsString(buffer[0 .. pos], wbuffer, WindowsStringConversionFlags.convertNewLines); + + DWORD written; + WriteConsoleW(hStdOut, toWrite.ptr, cast(DWORD) toWrite.length, &written, null); + } else { + DWORD written; + WriteFile(hStdOut, buffer.ptr, pos, &written, null); + } + } else { + import unix = core.sys.posix.unistd; + unix.write(1, buffer.ptr, pos); + } +} + +/+ + +STDIO + + /++ + Please note using this will create a compile-time dependency on [arsd.terminal] + + + +so my writeln replacement: + +1) if the std output handle is null, alloc one +2) if it is a character device, write out the proper Unicode text. +3) otherwise write out UTF-8.... maybe with a BOM but maybe not. it is tricky to know what the other end of a pipe expects... +[8:15 AM] +im actually tempted to make the write binary to stdout functions throw an exception if it is a character console / interactive terminal instead of letting you spam it right out +[8:16 AM] +of course you can still cheat by casting binary data to string and using the write string function (and this might be appropriate sometimes) but there kinda is a legit difference between a text output and a binary output device + +Stdout can represent either + + +/ + void writeln(){} { + + } + + stderr? + + /++ + Please note using this will create a compile-time dependency on [arsd.terminal] + + It can be called from a task. + + It works correctly on Windows and is user friendly on Linux (using arsd.terminal.getline) + while also working if stdin has been redirected (where arsd.terminal itself would throw) + + +so say you run a program on an interactive terminal. the program tries to open the stdin binary stream + +instead of throwing, the prompt could change to indicate the binary data is expected and you can feed it in either by typing it up,,,, or running some command like maybe <file.bin to have the library do what the shell would have done and feed that to the rest of the program + + +/ + string readln()() { + + } + + + // if using stdio as a binary output thing you can pretend it is a file w/ stream capability + struct File { + WritableStream ostream; + ReadableStream istream; + + ulong tell; + void seek(ulong to) {} + + void sync(); + void close(); + } + + // these are a bit special because if it actually is an interactive character device, it might be different than other files and even different than other pipes. + WritableStream stdoutStream() { return null; } + WritableStream stderrStream() { return null; } + ReadableStream stdinStream() { return null; } + ++/ + + +/+ + + +/+ + Druntime appears to have stuff for darwin, freebsd. I might have to add some for openbsd here and maybe netbsd if i care to test it. ++/ + +/+ + + arsd_core_init(number_of_worker_threads) + + Building-block things wanted for the event loop integration: + * ui + * windows + * terminal / console + * generic + * adopt fd + * adopt windows handle + * shared lib + * load + * timers (relative and real time) + * create + * update + * cancel + * file/directory watches + * file created + * file deleted + * file modified + * file ops + * open + * close + * read + * write + * seek + * sendfile on linux, TransmitFile on Windows + * let completion handlers run in the io worker thread instead of signaling back + * pipe ops (anonymous or named) + * create + * read + * write + * get info about other side of the pipe + * network ops (stream + datagram, ip, ipv6, unix) + * address look up + * connect + * start tls + * listen + * send + * receive + * get peer info + * process ops + * spawn + * notifications when it is terminated or fork or execs + * send signal + * i/o pipes + * thread ops (isDaemon?) + * spawn + * talk to its event loop + * termination notification + * signals + * ctrl+c is the only one i really care about but the others might be made available too. sigchld needs to be done as an impl detail of process ops. + * custom messages + * should be able to send messages from finalizers... + + * want to make sure i can stream stuff on top of it all too. + + ======== + + These things all refer back to a task-local thing that queues the tasks. If it is a fiber, it uses that + and if it is a thread it uses that... + + tls IArsdCoreEventLoop curentTaskInterface; // this yields on the wait for calls. the fiber swapper will swap this too. + tls IArsdCoreEventLoop currentThreadInterface; // this blocks on the event loop + + shared IArsdCoreEventLoop currentProcessInterface; // this dispatches to any available thread ++/ + + +/+ + You might have configurable tasks that do not auto-start, e.g. httprequest. maybe @mustUse on those + + then some that do auto-start, e.g. setTimeout + + + timeouts: duration, MonoTime, or SysTime? duration is just a timer monotime auto-adjusts the when, systime sets a real time timerfd + + tasks can be set to: + thread affinity - this, any, specific reference + reports to - defaults to this, can also pass down a parent reference. if reports to dies, all its subordinates are cancelled. + + + you can send a message to a task... maybe maybe just to a task runner (which is itself a task?) + + auto file = readFile(x); + auto timeout = setTimeout(y); + auto completed = waitForFirstToCompleteThenCancelOthers(file, timeout); + if(completed == 0) { + file.... + } else { + timeout.... + } + + /+ + A task will run on a thread (with possible migration), and report to a task. + +/ + + // a compute task is run on a helper thread + auto task = computeTask((shared(bool)* cancellationRequested) { + // or pass in a yield thing... prolly a TaskController which has cancellationRequested and yield controls as well as send message to parent (sync or async) + + // you'd periodically send messages back to the parent + }, RunOn.AnyAvailable, Affinity.CanMigrate); + + auto task = Task((TaskController controller) { + foreach(x, 0 .. 1000) { + if(x % 10 == 0) + controller.yield(); // periodically yield control, which also checks for cancellation for us + // do some work + + controller.sendMessage(...); + controller.sendProgress(x); // yields it for a foreach stream kind of thing + } + + return something; // automatically sends the something as the result in a TaskFinished message + }); + + foreach(item; task) // waitsForProgress, sendProgress sends an item and the final return sends an item + {} + + + see ~/test/task.d + + // an io task is run locally via the event loops + auto task2 = ioTask(() { + + }); + + + + waitForEvent ++/ + +/+ + Most functions should prolly take a thread arg too, which defaults + to this thread, but you can also pass it a reference, or a "any available" thing. + + This can be a ufcs overload ++/ + +interface SemiSynchronousTask { + +} + +struct TimeoutCompletionResult { + bool completed; + + bool opCast(T : bool)() { + return completed; + } +} + +struct Timeout { + void reschedule(Duration when) { + + } + + void cancel() { + + } + + TimeoutCompletionResult waitForCompletion() { + return TimeoutCompletionResult(false); + } +} + +Timeout setTimeout(void delegate() dg, int msecs, int permittedJitter = 20) { + return Timeout.init; +} + +void clearTimeout(Timeout timeout) { + timeout.cancel(); +} + +void createInterval() {} +void clearInterval() {} + +/++ + Schedules a task at the given wall clock time. ++/ +void scheduleTask() {} + +struct IoOperationCompletionResult { + enum Status { + cancelled, + completed + } + + Status status; + + int error; + int bytesWritten; + + bool opCast(T : bool)() { + return status == Status.completed; + } +} + +struct IoOperation { + void cancel() {} + + IoOperationCompletionResult waitForCompletion() { + return IoOperationCompletionResult.init; + } + + // could contain a scoped class in here too so it stack allocated +} + +// Should return both the object and the index in the array! +Result waitForFirstToComplete(Operation[]...) {} + +IoOperation read(IoHandle handle, ubyte[] buffer + +/+ + class IoOperation {} + + // an io operation and its buffer must not be modified or freed + // in between a call to enqueue and a call to waitForCompletion + // if you used the whenComplete callback, make sure it is NOT gc'd or scope thing goes out of scope in the mean time + // if its dtor runs, it'd be forced to be cancelled... + + scope IoOperation op = new IoOperation(buffer_size); + op.start(); + op.waitForCompletion(); ++/ + +/+ + will want: + read, write + send, recv + + cancel + + open file, open (named or anonymous) pipe, open process + connect, accept + SSL + close + + postEvent + postAPC? like run in gui thread / async + waitForEvent ? needs to handle a timeout and a cancellation. would only work in the fiber task api. + + waitForSuccess + + interrupt handler + + onPosixReadReadiness + onPosixWriteReadiness + + onWindowsHandleReadiness + - but they're one-offs so you gotta reregister for each event ++/ + + + +/+ +arsd.core.uda + +you define a model struct with the types you want to extract + +you get it with like Model extract(Model, UDAs...)(Model default) + +defaultModel!alias > defaultModel!Type(defaultModel("identifier")) + + + + + + + + + + +so while i laid there sleep deprived i did think a lil more on some uda stuff. it isn't especially novel but a combination of a few other techniques + +you might be like + +struct MyUdas { + DbName name; + DbIgnore ignore; +} + +elsewhere + +foreach(alias; allMembers) { + auto udas = getUdas!(MyUdas, __traits(getAttributes, alias))(MyUdas(DbName(__traits(identifier, alias)))); +} + + +so you pass the expected type and the attributes as the template params, then the runtime params are the default values for the given types + +so what the thing does essentially is just sets the values of the given thing to the udas based on type then returns the modified instance + +so the end result is you keep the last ones. it wouldn't report errors if multiple things added but it p simple to understand, simple to document (even though the default values are not in the struct itself, you can put ddocs in them), and uses the tricks to minimize generated code size ++/ + ++/ + +private version(Windows) extern(Windows) { + BOOL CancelIoEx(HANDLE, LPOVERLAPPED); + + struct WSABUF { + ULONG len; + ubyte* buf; + } + alias LPWSABUF = WSABUF*; + + // https://learn.microsoft.com/en-us/windows/win32/api/winsock2/ns-winsock2-wsaoverlapped + // "The WSAOVERLAPPED structure is compatible with the Windows OVERLAPPED structure." + // so ima lie here in the bindings. + + int WSASend(SOCKET, LPWSABUF, DWORD, LPDWORD, DWORD, LPOVERLAPPED, LPOVERLAPPED_COMPLETION_ROUTINE); + int WSASendTo(SOCKET, LPWSABUF, DWORD, LPDWORD, DWORD, const sockaddr*, int, LPOVERLAPPED, LPOVERLAPPED_COMPLETION_ROUTINE); + + int WSARecv(SOCKET, LPWSABUF, DWORD, LPDWORD, LPDWORD, LPOVERLAPPED, LPOVERLAPPED_COMPLETION_ROUTINE); + int WSARecvFrom(SOCKET, LPWSABUF, DWORD, LPDWORD, LPDWORD, sockaddr*, LPINT, LPOVERLAPPED, LPOVERLAPPED_COMPLETION_ROUTINE); +} |