I cannot delete the language I downloaded

[suatsuphi]

Hi,

It gives me an error when I delete the language I downloaded from Available Languages in the Languages section.

I can delete the language I added manually.

https://ibb.co/qpnBGgB

[mgesing]

I cannot reproduce this problem. Sqlite.Exception "database disk image is malformed" sounds like a corrupted database file.

[suatsuphi]

If I run the shrink (admin/maintenance/systeminfo/) , I can delete it.
var sql = $"VACUUM;PRAGMA wal_checkpoint=TRUNCATE;PRAGMA optimize;PRAGMA wal_autocheckpoint;";

if I can delete the language I added manually. this situation does not occur...

[muratcakir]

@mgesing Please check again with SQLite. Could be a file locking problem caused by the fire-forget async operation.

[mgesing]

I still cannot reproduce it. Of course I have tried it with SQLite.

[mgesing]

Does this happen only with a specific language or with all downloaded languages?

[suatsuphi]

yes, I all downloaded languages.

• I downloaded the project from github.
• I ran the project.
• I did the installation in English (sqlite)
• I downloaded Turkish from admin/language/list/.
• I made the default language and the language of the admin panel Turkish. The language of the admin panel must be Turkish.
  https://ibb.co/vPjR8tM
• I downloaded French from admin/language/list/.
• Then it gives me an error when I try to delete the French I downloaded.

note: as I understand it that when I use the admin panel in Turkish, it does not allow me to delete the languages which I downloaded? allows me to delete languages that I have manually saved (admin/language/create/).

=================

I just realized now that this problem does not occur if shrink is run once. Now I download and delete and no error occurs.

I made Turkish the second language and then updated it with the refresh button. then I made it the first language again and made the admin panel language Turkish. when I add a new language by download, this error occurs again

[mgesing]

Looks like the reason is a corrupted index on LocaleStringResource.

PRAGMA integrity_check;

gave me a hint about it.

PRAGMA writable_schema=ON;
VACUUM;

fixed it for me. (It cannot be excuted via backend, only via SQLite Db Browser.) Had to switch back from Turkish to English to get the language deleted afterwards. How can a database system be so fragile....

[suatsuphi]

https://ibb.co/Q8tRBqk
You cannot delete a language you downloaded in screen language when you change the language. for example you downloaded a language in English screen language. if you switch to German or French, you get this index error

[suatsuphi]

I figured it out... problem is here >> //var xmlResourceManager = scope.Resolve<IXmlResourceManager>();

I removed the DownloadCore method from being static - to be able to use the _xmlResourceManager from ctor

now it is working...

        private async Task DownloadCore(ILifetimeScope scope, LanguageDownloadContext context, CancellationToken cancelToken)
        {
            var appContext = scope.Resolve<IApplicationContext>();
            var asyncState = scope.Resolve<IAsyncState>();
            var httpClientFactory = scope.Resolve<IHttpClientFactory>();
            //var xmlResourceManager = scope.Resolve<IXmlResourceManager>();
            var db = scope.Resolve<SmartDbContext>();

            try
            {
                // 1. Download resources.
                var state = new LanguageDownloadState
                {
                    Id = context.SetId,
                    ProgressMessage = context.StringResources["Admin.Configuration.Languages.DownloadingResources"]
                };

                await asyncState.CreateAsync(state, null, false, CancellationTokenSource.CreateLinkedTokenSource(cancelToken));

                var client = httpClientFactory.CreateClient();
                var resources = context.AvailableResources.Resources.First(x => x.Id == context.SetId);
                var xmlDoc = await DownloadAvailableResources(client, resources.DownloadUrl, context.StoreUrl, cancelToken);

                if (!cancelToken.IsCancellationRequested)
                {
                    using var dbScope = new DbContextScope(db, minHookImportance: HookImportance.Essential);
                    await asyncState.UpdateAsync<LanguageDownloadState>(state => state.ProgressMessage = context.StringResources["Admin.Configuration.Languages.ImportResources"]);

                    var cultureCode = resources.Language.Culture;

                    // 2. Create language entity (if required).
                    var language = await db.Languages
                        .Where(x => x.LanguageCulture == cultureCode)
                        .FirstOrDefaultAsync(cancelToken);

                    if (language == null)
                    {
                        var maxDisplayOrder = await db.Languages
                            .Where(x => x.Published)
                            .MaxAsync(x => (int?)x.DisplayOrder, cancelToken);

                        language = new Language
                        {
                            LanguageCulture = cultureCode,
                            UniqueSeoCode = resources.Language.TwoLetterIsoCode,
                            Name = GetCultureDisplayName(cultureCode) ?? resources.Name,
                            FlagImageFileName = GetFlagFileName(cultureCode, appContext),
                            Rtl = resources.Language.Rtl,
                            Published = false,
                            DisplayOrder = maxDisplayOrder.HasValue ? maxDisplayOrder.Value + 1 : 0
                        };

                        db.Languages.Add(language);
                        await dbScope.CommitAsync(cancelToken);
                    }

                    // 3. Import resources.
                     await _xmlResourceManager.ImportResourcesFromXmlAsync(language, xmlDoc);

                    // 4. Save import info.
                    var serializedImportInfo = JsonConvert.SerializeObject(new LastResourcesImportInfo
                    {
                        TranslatedPercentage = resources.TranslatedPercentage,
                        ImportedOn = DateTime.UtcNow
                    });

                    var attribute = await db.GenericAttributes.FirstOrDefaultAsync(x =>
                        x.Key == "LastResourcesImportInfo" &&
                        x.KeyGroup == nameof(Language) &&
                        x.EntityId == language.Id &&
                        x.StoreId == 0, cancelToken);

                    if (attribute == null)
                    {
                        db.GenericAttributes.Add(new()
                        {
                            Key = "LastResourcesImportInfo",
                            KeyGroup = nameof(Language),
                            EntityId = language.Id,
                            StoreId = 0,
                            Value = serializedImportInfo
                        });
                    }
                    else
                    {
                        attribute.Value = serializedImportInfo;
                    }

                    await dbScope.CommitAsync(cancelToken);
                }
            }
            catch (Exception ex)
            {
                scope.Resolve<ILogger>().ErrorsAll(ex);
            }
            finally
            {
                if (asyncState.Contains<LanguageDownloadState>())
                {
                    asyncState.Remove<LanguageDownloadState>();
                }
            }
        }

[muratcakir]

I removed the DownloadCore method from being static - to be able to use the _xmlResourceManager from ctor

But now you will have to deal with possible memory leaks, because DownloadCore is not awaited (fire-forget). The request ends before the method completes execution. The controller needs to be GCed, but DownloadCore is still using resources from its outer scope (the controller).

Better we investigate what's going on in IXmlResourceManager.

[mgesing]

@suatsuphi Please test if the problem was fixed by b9110c5 for you, too.

[suatsuphi]

it is working now with the patch that you made.

The index problem occurs when downloading, not deleting. The reason for this seems to be xmlResourceManager. Calling from xml ctor as below solves the problem. No index problem occurs as below.

https://ibb.co/LYMhm4M

[Permission(Permissions.Configuration.Language.EditResource)]
public async Task<IActionResult> Download(int setId)
{
    var resources = await CheckAvailableResources();

    if (resources.Resources.Count > 0)
    {
        var ctx = new LanguageDownloadContext(setId)
        {
            AvailableResources = resources,
            StoreUrl = Services.StoreContext.CurrentStore.GetBaseUrl(),
            StringResources = new()
            {
                { "Admin.Configuration.Languages.ImportResources", T("Admin.Configuration.Languages.ImportResources") },
                { "Admin.Configuration.Languages.DownloadingResources", T("Admin.Configuration.Languages.DownloadingResources") }
            }
        };

        _ = _asyncRunner.RunTask((scope, ct, state) => DownloadCore(_xmlResourceManager, scope, state as LanguageDownloadContext, ct), ctx);
    }

    return RedirectToAction(nameof(List));
}

private static async Task DownloadCore(IXmlResourceManager xmlResourceManager,ILifetimeScope scope, LanguageDownloadContext context, CancellationToken cancelToken)
{
    var appContext = scope.Resolve<IApplicationContext>();
    var asyncState = scope.Resolve<IAsyncState>();
    var httpClientFactory = scope.Resolve<IHttpClientFactory>();
    //var xmlResourceManager = scope.Resolve<IXmlResourceManager>();
    var db = scope.Resolve<SmartDbContext>();
...

XmlResourceManager.cs : I have culture in XmlResourceManager - even if I make the value here in English the same problem occurs. Like the exact solution above...

public partial class XmlResourceManager : IXmlResourceManager
{
    [GeneratedRegex("^resources.(.+?).xml$", RegexOptions.IgnoreCase | RegexOptions.Compiled, "de-DE")]
    private static partial Regex FileNameRegEx();

[mgesing]

You do not understand scoping. Your code does not insert any string resource. That's why the database index does not get corrupted and why there is no "database disk image is malformed".

[suatsuphi]

you are right...

but I think the problem is the XmlResourceManager.
Could I use CurrentCulture in the GeneratedRegex structure?

https://ibb.co/FmhPQny

[mgesing]

I changed this 271a052. This is Intellisense suggested and auto generated code from yesterday commit. If its German or invariant culture should not make any difference in this particular case. I'm surprised that bothers you.

[suatsuphi]

I thought the problem was with culture. I think I found out what the problem was. I tested it locally and it works fine.
https://ibb.co/3dHvygm

// 3. Import resources. 
//await xmlResourceManager.ImportResourcesFromXmlAsync(language, xmlDoc);
// for test
var xmlDoc2 = XDocument.Load(@"C:\Users\FB\Downloads\smartstore-string-resources-5.0.5.0-de-de.xml");
await xmlResourceManager.ImportResourcesFromXmlAsync(language, xmlDoc2.ToString());

I need to fix two ResourceValue values in all languages so that I can test them by downloading. otherwise it causes indexing and then malformed error.

	<LocaleResource Name="admin.system.systeminfo.appversion">
		<Value>Smartstore version</Value>
	</LocaleResource>
	<LocaleResource Name="admin.system.systeminfo.appversion.hint">
		<Value>Smartstore version</Value>
	</LocaleResource>

I think the problem will be solved when we fix the "info" here in all languages. Because the "i" character is not equal to the "I" character.

Can you fix this problem in xml files?

[mgesing]

Interesting, but this looks like a workaround. There are very many resource names that are lowercase and contain an i. Should they all be changed manually, for all 32 languages?

[suatsuphi]

No, just this tag > systeminfo, because the use is different
just two record (LocaleResource ) all language record...
the problem arose because these two records were written in lower case. this is enough systemInfo
https://ibb.co/L9tYkdD

like

	<LocaleResource Name="Admin.System.SystemInfo.Appversion">
		<Value>Smartstore version</Value>
	</LocaleResource>
	<LocaleResource Name="Admin.System.SystemInfo.Appversion.Hint">
		<Value>Smartstore version</Value>
	</LocaleResource>

[mgesing]

Why these two? Why is the usage different? The XML contains a lot of names in lower case and LocalizedDisplay is always in pascal case. So the case applies to many string resources.

[suatsuphi]

Why these two?

because these two records are not pascal case

The XML contains a lot of names in lower case and LocalizedDisplay is always in pascal case.

where is the converter that LocalizedDisplay is always in pascal case? Maybe something can be done here?

So the case applies to many string resources.

No, only pascal case applies to unused 'LocaleResource - Name' and just thesee two records
admin.system.systeminfo.appversion
admin.system.systeminfo.appversion.hint
All other entries are written within Pascal Case rules. I checked them.

We had a similar problem here.
#714
It was fixed when the related records were made pascal case.

[mgesing]

There are more than the two resources in lower case. These are very old resources that have been in the XML file from the beginning, not added later. I have fixed Admin.System.SystemInfo.AppVersion via migration now. But this is just the beginning. When you download Turkish you still get the lower case version though. I don't want to change it manually in Smartstore Translate. But you will get it after the next published Smartstore version when it will be changed there too. Smartstore Translate routinely updates the resources a short time after a new Smartstore version has been published.

[suatsuphi]

thank you

I looked at the records and there are only 14 records with lowercase typed - pascal case does not fit.

account.login.newcustomertext
admin.appnews
admin.configuration.deliverytime.added
admin.configuration.deliverytime.deleted
admin.configuration.plugins.description.step1
admin.configuration.themes.option.defaultdesktoptheme.hint
admin.configuration.themes.option.defaultmobiletheme.hint
admin.orders.pdfinvoice
admin.pagetitle
admin.system.systeminfo.appversion
admin.system.systeminfo.appversion.hint
admin.system.warnings.incompatibleplugin
order.getpdfinvoice
products.callforprice

only two of these records are causing problems.

admin.system.systeminfo.appversion
admin.system.systeminfo.appversion.hint

the reason why these are causing problems is that the pascal case of the previous admin.system.systeminfo info is set differently

Admin.System.SystemInfo
Admin.System.SystemInfo
Admin.System.SystemInfo.AppDate
Admin.System.SystemInfo.AppDate
Admin.System.SystemInfo.AppDate.Hint
Admin.System.SystemInfo.AppDate.Hint
Admin.System.SystemInfo.ASPNETInfo
Admin.System.SystemInfo.ASPNETInfo
Admin.System.SystemInfo.ASPNETInfo.Hint
Admin.System.SystemInfo.ASPNETInfo.Hint
Admin.System.SystemInfo.DatabaseSize
Admin.System.SystemInfo.DatabaseSize
Admin.System.SystemInfo.DatabaseSize.Hint
Admin.System.SystemInfo.DatabaseSize.Hint
Admin.System.SystemInfo.DataProviderFriendlyName
Admin.System.SystemInfo.DataProviderFriendlyName
Admin.System.SystemInfo.DataProviderFriendlyName.Hint
Admin.System.SystemInfo.DataProviderFriendlyName.Hint
Admin.System.SystemInfo.DbTableInfo
Admin.System.SystemInfo.DbTableInfo
Admin.System.SystemInfo.DbTableInfo.NumRows
Admin.System.SystemInfo.DbTableInfo.NumRows
Admin.System.SystemInfo.DbTableInfo.TableName
Admin.System.SystemInfo.DbTableInfo.TableName
Admin.System.SystemInfo.DbTableInfo.TotalSpace
Admin.System.SystemInfo.DbTableInfo.TotalSpace
Admin.System.SystemInfo.DbTableInfo.UnusedSpace
Admin.System.SystemInfo.DbTableInfo.UnusedSpace
Admin.System.SystemInfo.DbTableInfo.UsedSpace
Admin.System.SystemInfo.DbTableInfo.UsedSpace
Admin.System.SystemInfo.GarbageCollect
Admin.System.SystemInfo.GarbageCollect
Admin.System.SystemInfo.GarbageCollectSuccessful
Admin.System.SystemInfo.GarbageCollectSuccessful
Admin.System.SystemInfo.HTTPHOST
Admin.System.SystemInfo.HTTPHOST
Admin.System.SystemInfo.HTTPHOST.Hint
Admin.System.SystemInfo.HTTPHOST.Hint
Admin.System.SystemInfo.IsFullTrust
Admin.System.SystemInfo.IsFullTrust
Admin.System.SystemInfo.IsFullTrust.Hint
Admin.System.SystemInfo.IsFullTrust.Hint
Admin.System.SystemInfo.LoadedAssemblies
Admin.System.SystemInfo.LoadedAssemblies
Admin.System.SystemInfo.LoadedAssemblies.Hint
Admin.System.SystemInfo.LoadedAssemblies.Hint
Admin.System.SystemInfo.OperatingSystem
Admin.System.SystemInfo.OperatingSystem
Admin.System.SystemInfo.OperatingSystem.Hint
Admin.System.SystemInfo.OperatingSystem.Hint
Admin.System.SystemInfo.ServerLocalTime
Admin.System.SystemInfo.ServerLocalTime
Admin.System.SystemInfo.ServerLocalTime.Hint
Admin.System.SystemInfo.ServerLocalTime.Hint
Admin.System.SystemInfo.ServerTimeZone
Admin.System.SystemInfo.ServerTimeZone
Admin.System.SystemInfo.ServerTimeZone.Hint
Admin.System.SystemInfo.ServerTimeZone.Hint
Admin.System.SystemInfo.UsedMemorySize
Admin.System.SystemInfo.UsedMemorySize
Admin.System.SystemInfo.UTCTime
Admin.System.SystemInfo.UTCTime
Admin.System.SystemInfo.UTCTime.Hint
Admin.System.SystemInfo.UTCTime.Hint
admin.system.systeminfo.appversion
admin.system.systeminfo.appversion.hint

Admin.System.SystemInfo > where the letters S and I are capitalized

but for the appversion resourcename it stays as Systeminfo. So it creates a different directory and this creates a problem. I don't understand at what stage this occurs, but I am sure it is related to the cache because I manually delete the relevant records and if I don't delete the cache, it gives an error. So I have to delete the cache. There is a situation related to the pascal case between the cache and the database beacuse SystemInfo is not equal Systeminfo

[mgesing]

admin.system.systeminfo.appversion has been changed. admin.system.systeminfo.appversion.hint no longer exists (because it has the same value). So it is fixed now. Despite note on Smartstore Translate. And let symptoms be symptoms.