Some thoughts about modularization

  • Hi
    this should be a clarification threat to some thoughs we wrote on IRC last night....
    there were some mssunderstandings, so I try to rewrite the thoughts...


    We could define:
    - the i-MSCP Gui only reads&writes to the i-mscp database and does not need full MySQL root privileges. Customer passwords (mail, ftp, mysql, webfolder protection etc etc) are stored encypted - always.
    - for every service an engine script reads the relevant data from the i-mscp database, changes them as neede (e.g. makes hashes, rearrages the data...) and writes the data for productive operation to the needed places (depends on the type and package of service). If the service can use a database, then maybe it's possible to use an i-MSCP-prod database for theese data (so it's not inside the internal i-mscp db). And remember: also creation of a customer database & db user should be done by a engine script (not by the i-MSCP GUI).


    This makes it possible to have the engine scripts running on different servers - a step closer to the *MS* in i-MSCP....
    and it's easier to adopt i-MSCP to other services (eg. dovecot instead of courier etc...) or to add new modules (eg. cron :-)


    I hope it's more clear now what kassah and I wanted to tell....


    /Joxi

  • Just for the record, this poses a problem about the (few) cases where the gui must access a domains' files (to choose the webroot for example). For now, the virtual file system class creates a temporary FTP user and uses it to access the domain's files. Nevertheless, this can not be done if there's the "round-trip" of having to call the backend to create such user. Therefore, if we get to the point of creating ftp users trhough the backend, we must also implement a "permanently hidden for-gui-usage-only ftp user" that is created along with the domain and never gets deleted.

  • Right killburn. Maybe it's possible to create such a ftp user with each domain or with each user (depending on the system users and permissions) - and then also configure the ftp server to only accept theese userlogins from "localhost" (or from the adminpanel-Server).


    /J


  • - the i-MSCP Gui only reads&writes to the i-mscp database and does not need full MySQL root privileges


    We need to add more functionality to daemon in order to bypass full root privs. i.e. some way to ask the daemon "Does this database exist?" before we schedule it.



    For now, the virtual file system class creates a temporary FTP user and uses it to access the domain's files. Nevertheless, this can not be done if there's the "round-trip" of having to call the backend to create such user.


    I ran into this issue when doing net2ftp integration, we need some kind of gui accessible FTP. (scp/sftp with pub key auth maybe?)

  • As soon as i-MSCP runs on another Server than the services (the *MS*) then all the direct mechanisms of the actual ispcp are useless.
    so we have to do it via daemon (or something which replaces the daemon...) which runs on every of the controlled server...


    That's also why the productive mail-users need to be in a separate DB (not on the i-MSCP-Server but on the Mailserver...) :-)


    Seems that we need a good communication between the gui->daemon->servers... etc


    and we need to think different than before...


    /Joxi

  • Hello friends ,


    First, sorry for the delay:


    I post here a little discussion that is linked to the both Joximu and kassah proposals:


    <nuxwin> I'm very tired with the current version
    <nuxwin> very shit
    <nuxwin> we will adopt joximu idea to start new version from scratch
    <nuxwin> to resume
    <nuxwin> data are added in GUI database but
    <sci2tech> engine do shit, gui just write in database?
    <nuxwin> GUI do nothing more
    <nuxwin> yes
    <nuxwin> with event handler for queue job
    <nuxwin> sound like scheduled tasks for each servers
    <sci2tech> ok from me. but there are some problems with this ftp as sql management. but can be done
    <nuxwin> I explain to you, imagine the following:
    <nuxwin> I've only GUI and database on one server
    <nuxwin> on other server, I've services such as Apache, php, ftp..
    <nuxwin> and on other, I've Database server for customers
    <nuxwin> and on other, I've mail service ....
    <nuxwin> Imagine now I adds a mail account from the GUI
    <nuxwin> then, the GUI will just adds the relevant data in GUI in normalized way
    <nuxwin> and will add ADD_MAIL event in queue
    <nuxwin> and then, he will notify all servers
    <nuxwin> then, all servers will look at the queue to see if he have awaiting task
    <nuxwin> and do it if yes
    <nuxwin> So, what you think about that ?
    <nuxwin> Hard to explain but I think you got the idea
    <nuxwin> so on, for example, the mail server will run this own engine part for mail
    <sci2tech> yes
    <nuxwin> take the data in the database
    <nuxwin> and then, will add the account data like he want
    <nuxwin> About customer Database now
    <nuxwin> the GUI will only add relevant data for new database and user in normalized way too
    <nuxwin> but will not create the database and user
    <nuxwin> instead of that, he will adds event in queue
    <nuxwin> such as ADD_SQLD and will notify the servers for event
    <nuxwin> and then, the server for database will do the job
    <nuxwin> but ok, still the problem of SQL users linked to the databases
    <nuxwin> BTW, we should fin a way to link user to customers databases
    <nuxwin> but you got the idea ?
    <nuxwin> by doing like this, for example, if one user want not use the daemon at all, h ecan!
    <nuxwin>I mean, he can just add cron task on each servers to look at event queue sometime, and then do the job if one is scheduled...
    <nuxwin> you got the idea ?
    <sci2tech> yes
    <sci2tech> but for service
    <nuxwin> what you don't understand ?
    <sci2tech> Mysql will be same for all servers?
    <nuxwin> in fact Mysql for the ispCP GUI database yes
    <nuxwin> but with the idea we can have more than one mysql server for customer's databases and also, service such as Proftpd can have this own database on main Mysql server or another server..
    <nuxwin> but right all services server will take data and see queue from same SQL server
    <nuxwin> btw: (The Mysql server that host the GUI database)
    <sci2tech> if all servers take data from same server for ftp
    <nuxwin> what about ftp
    <sci2tech> ftp read data directly from db for connection
    <nuxwin> ok I explain
    <nuxwin> for ftp
    <sci2tech> why wait engine to add this data when GUI can do it faster?
    <nuxwin> because with that user can use easily other ftp server than proftpd
    <nuxwin> I explain
    <nuxwin> 1. The GUI will only store username and password and mount_point for ftp account in normalized way
    <nuxwin> and will add event such as ADD_FTP in queue
    <nuxwin> then, the server in charge of FTP service will look at the queue and see a task for him and then, he will launch this own script to adding ftp account
    <nuxwin> he will transform data FROM GUI like he want. He can store it in flat file or in specific database (like he want...). That allow also to use other Ftp server than proftpd as long plugin for engine is provided. Same concept for all other services sucha as postfix, courier....
    <sci2tech> ah ok
    <sci2tech> right
    <sci2tech> this way we can use server that do not use Mysql ;)
    <nuxwin> yes
    <sci2tech> this way we can use ftp server that do not use Mysql ;) but plain file
    <sci2tech> ok agree with that
    <nuxwin> yes
    <sci2tech> also we can split databases over multiple Mysql servers
    <nuxwin> yes
    <sci2tech> ok. joximu proposal has my support
    <nuxwin> hehe
    <nuxwin> Will report our discussion
    <nuxwin> in the forum to show joxi
    <sci2tech> great
    <nuxwin> we have understand
    <nuxwin> I've only add the kassah event handler proposal here
    <sci2tech> great
    <sci2tech> love this 2 guys
    <nuxwin> yeah

    badge.php?id=1239063037&bid=2518&key=1747635596&format=png&z=547451206

  • Hi nuxi and sci2tech
    great you like the idea... I wanted to separate the things since a long time but with the multiple server thing it's now more interesting and makes more sense... :-)


    I don't understand this part:

    Quote


    <nuxwin> but ok, still the problem of SQL users linked to the databases
    <nuxwin> BTW, we should fin a way to link user to customers databases
    <nuxwin> but you got the idea ?
    <nuxwin> by doing like this, for example, if one user want not use the daemon at all, h ecan!


    what do you mean with "link users to customers database" or "if a user don't want to use the daemon, he can"... ?


    let's start to make some drafts... maybe the wiki has a place where we could discuss this in a written way...


    /Joxi

  • I mostly agree with nuxwin's proposal. Nevertheless, I feel there's a missing piece in there.


    Essentially, what happens if there are multiple servers that have to act on an "add mail" task in the queue? For instance, consider a situation where someone has:


    (1) A mail gateway that does spam filtering
    (2) A mail server that stores mails for end users (offering pop3/imap services)


    When a new mail account is added, both servers have to update its configuration to enable the new e-mail address. Now imagine that one of the servers succeeds in adding this new mail account whereas the other one fails. How do they both report their success/failure to a single task? The gui database is doomed to only show one of their statuses.


    Here's where "transaction based" tasks kick in. Essentially, there should be some "task dispatcher" who knows that both server A and server B should update their configs in order for the new e-mail address to become functional. Then, this "task dispatcher" would be the responsible for launching a "transaction" that comprises both tasks (updating the configs on server 1 *and* updating the configs on server 2). Obviously, the actual tasks (that compose the transaction) can be also stored in the database just like in nuxwin's proposal. Then, each server would report its success and failure respectively. Thereafter, the "task dispatcher" would know that one of the servers has failed, and would tell the other one to "rollback", while updating the "add e-mail transaction" to a failed status.


    This way we could keep track of single operations that affect multiple servers at once, while maintaining a "stable" state when something fails. IMHO this is a basic requirement for a real multi-server control panel...


    Comments?

  • killburn - right:


    we should think about this - but we do not need to start with this scenario...


    /J

    Edited once, last by joximu ().

  • Well we already have the issue kilburn is trying to solve, instead of multiple servers we have multiple config. For instance, when a domain is added, it has to be added to apache, mail and dns. If any one of those fail, we currently just get stuck. (Doesn't happen often thankfully)